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This publication features work presented at the Second 
Annual Virtual Reference Desk Digital Reference Conference (October 6-17, 
2000, Seattle, Washington) . These proceedings include papers and 
presentations by conference presenters representing public, academic, and 
government libraries, as well as subject-specific AskA services, from the 
United States, Canada, Australia, Denmark, and Japan. Conference participants 
identified several key issues facing practitioners and researchers. The 
publication opens with records from plenary sessions, including the "Welcome'’ 
presentation by R. David Lankes; transcription from the keynote address by 
Michael B. Eisenberg and Charles R. McClure; and background on the "Impacts 
of Digital Reference" panel. The presentations and papers that follow provide 
a snapshot of current services, research initiatives, and products that help 
define the field of digital reference. Sections are arranged by topic (or 
conference track) , each one highlighting a different area of digital 
reference: (1) "Real-Time Technologies" features services using Web contact 

software and instant messaging technologies for digital reference; (2) 
"Software for Digital Reference" introduces tools developed in-house for 
specific services as well as customizable applications; (3) "Managing Digital 
Reference Services" discusses issues in service development and maintenance 
including staffing, question-answer procedures, publicity, technical support, 
and modes of service; (4) "Issues and Research in Digital Reference" presents 
a range of topics including standards development, commercial versus 
non-profit services, communications issues, and digital reference service in 
government and distance education contexts; (5) "Spotlight on User Needs and 
Behaviors" focuses on methods for gathering and interpreting data from users 
and translating needs into effective service; (6) "Digital Reference Service 
Spotlights" includes first-hand experiences from one sub j ect- related AskA 
service and academic and government libraries; (7) "Digital Reference 
Networks" offers insights on collaborative efforts in digital reference among 
multiple institutions; and (8) "Resources in Digital Reference" highlights 
processes for developing, procuring, and evaluating resources for use in 
online and traditional and reference service. (AEF) 
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Preface 



This publication features work presented at the Second Annual Virtual Reference Desk 
Digital Reference Conference, “Facets of Digital Reference,” held October 16-17, 2000 
in Seattle, WA. These proceedings include papers and presentations by conference 
presenters representing public, academic and government libraries, as well as subject- 
specific AskA services, from the United States, Canada, Australia, Denmark, and Japan. 

Conference participants identified several key issues facing practitioners and researchers 
including scalability, or the ability of services to grow exponentially in response to user 
demands; quality criteria for expert responses and evaluation methods; the proliferation 
of new commercial services and increased competition for libraries; and the use of new 
software technologies and tools to help automate and streamline Internet-based 
information exchanges. 

The publication opens with records from conference plenary sessions, including the 
“Welcome” presentation by R. David Lankes; transcription from the keynote address by 
Michael B. Eisenberg and Charles R. McClure (“Digital Reference Librarians: Who 
Needs ‘Em?”); and background on the discussion and participants of the “Impacts of 
Digital Reference” panel. 

The presentations and papers that follow provide a snapshot of current services, research 
initiatives, and products that help define the field of digital reference. Sections are 
arranged by topic (or conference track), each one highlighting a different area of digital 
reference: 

1 . “Real-Time Technologies” features services using Web contact software and 
instant messaging technologies for digital reference; 

2. “Software for Digital Reference” introduces tools developed in-house for specific 
services as well as customizable applications; 

3. “Managing Digital Reference Services” discusses issues in service development 
and maintenance including staffing, question-answer procedures, publicity, 
technical support, and modes of service; 

4. “Issues and Research in Digital Reference” presents a range of topics including 
standards development, commercial vs. non-profit services, communications 
issues, and digital reference service in government and distance education 
contexts; 

5. “Spotlight on User Needs and Behaviors” focuses on methods for gathering and 
interpreting data from users and translating needs into effective service; 

6. “Digital Reference Service Spotlights” includes first-hand experiences from one 
subject-related AskA service and academic and government libraries; 

7. “Digital Reference Networks” offers insights on collaborative efforts in digital 
reference among multiple institutions; and 

8. “Resources in Digital Reference” highlights processes for developing, procuring, 
and evaluating resources for use in on-line and traditional reference service. 
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The purpose of the proceedings is to provide information professionals and digital 
reference service providers with the most up-to-date information in this quickly growing 
field and to allow institutions to share thoughts and experiences with each other. The 
editors hope that the information presented here will contribute to the ongoing dialog on 
the subject of digital reference and to efforts in the development of quality and technical 
standards. 

This publication is also available on the Virtual Reference Desk Web site at: 
http://vrd.org/conferencesWRD2000/proceedings/index.shtml 
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Digital Reference Librarians: Who Needs 'Em? 

Or 

Key Issues in Digital Reference: Taking Action 



Michael B. Eisenberg, 

Director and Professor, School of Library & Information Science, University of Washington, 

Seattle 

Charles R. McClure, 

Francis Eppes Professor, School of Information Studies, Florida State University, and 
Director, Information Use Management and Policy Institute 



INTRODUCTION 

Mike Good morning, hope you can see us ok. Chuck and I decided that we do lots of speeches 
and presentations, but what we don't get to do anymore is talk. Chuck and I, for 1 5 years 
or 14 years, had offices right next to each other at Syracuse University in the School of 
Information Studies, and we could just knock on the wall, and say, "Shut up", or, "I 
disagree with that," or whatever. And, Chuck would get passionate on the phone, and 
Chuck loved to hold telephone conference calls with his speakerphone out there. And I 
would hear the whole thing, and I would disagree through the wall. Now that Chuck is in 
Florida, and I'm in Seattle, which, think about it folks, we got about as far away from 
each other as you can in the continental United States! 

Chuck There is a reason for that, too, by the way. 

Mike When Dave and the group at the Information Institute asked us to participate in the 

Facets of Digital Reference Conference, we thought what might be nice is for us to have 
a conversation. And so we have come up with an outline, but we have to tell you that a 
lot of this is going to be a bit free form - not that we don't have fairly well thought out 
and documented thoughts about this area, but we thought it would be more of a 
conversation. And we left an empty chair here, in case any of you were so burning with 
the desire, that you've got to jump up and say something, even during the interaction, so 
the chair is open. So, Dave gave this rousing intro speech, but I think Chuck and I, if [we] 
can be so bold, might say, "Maybe we're not so sure." All this great VRD stuff, and all 
this new VRD stuff, whatever, . . .ah. . . let's take a closer look at the issues. 

Chuck Does Dave get this emotionally upset all the time? "Do it, do it. . . " What is it? 

Mike Two things: he's on east coast time. . .. 

Chuck Is that it? 

Dave If I bring up a couch, can we get therapy? 

Mike Get out of here.. . So, what we really want here is to take stock, and stop and think. In 
some ways, Dave may be right, because we are on the cusp of a reference revolution. So 
when the Library of Congress (read about [their project in] American Libraries ), is 
talking about 24/7 global reference services- that's pretty interesting stuff. But, maybe 
we better think about where we're going, and [if] digital reference [has] really arrived. Is 
it still viable? In the forms that we are talking about? Other alternatives that might better 
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serve such needs. And what are the opportunities, [and] some of the cautions? And what 
are the various directions are that we might take? And lastly, if digital reference has 
finally made it, maybe we really better think about it. Because it's like that Woody Allen 
story, that I never want to belong to a club that would want someone like me in it, right? 
Well, if the world has said [that] digital reference is a really good idea, maybe we better 
be thinking about the next thing. Because they are not always right. So, if everyone in 
the world, except maybe Chuck, thinks that digital reference is a good idea, is it really? 

Chuck Well, you know, I really have to laugh, listening to Dave earlier. He's like, ‘this 

could be a really good thing, the number of traditional reference questions going down in 
our library. That gives us more time to prepare for the digital onslaught.’ Excuse me 
very much, ya'll. Do you like that, “ya'll?” One year in Florida, boom, and you're ruined. 
But the fact of the matter is, while those statistics are going down at the reference desk, 
we don't have any way to figure out [why] the stats are going up on the Web site, or on 
the digital reference site. So, what I'm trying to get to today, is that I think there are a 
number of issues that we need to start thinking about. And, some of these issues, I think, 
are fairly serious. Now, one of the problems here, Mike, you know, is that we got 
converted. It was great, you know, Dave's going, "And we're number one," I was ready 
for you guys [to] go, "Put me in coach, put me in — I'm ready!" The point here is, Mike, 
that we have a bunch of converted people sitting out here. They already believe. 

Mike Well, the reason they believe is because digital reference has grown up over the last 5 to 
8 to 10 years. From some fairly humble beginnings, but it has really kind of made it. I 
remember, I was there, as Dave said, when we started The Electronic Librarian, TEL, 
which evolved into AskERIC. And it was Dave that helped us to set up the first reason 
that we thought we needed that. There was this wonderful new tool out there called 
Gopher. Remember Gopher? And we had put together a team at the ERIC 
Clearinghouse; with support we had recreated what we thought [was] the best Gopher site 
and system in the entire world, and we had invested thousands in people hours in learning 
how to do that. Except, when we put this Gopher stuff up, nobody could find anything. 
So, other people were inventing things like Veronica. Remember Veronica? Remember 
the first time? The librarians in the room: you remember? It was an index, Chuck. In a 
lot of my speeches, I have librarians say the word., .say that, say the word, "index." 

Audience Index 

Mike Get all tingly, feel good, whatever, right? We're in the right room, Chuck. That's what 
librarians do, we love and relate to indexes. 

Chuck We're dealing with some sick people out there. 

Mike It's actually genetic, we've done this study at the University of Washington Medical 
Center. 

Chuck I don't want to know. 

Mike In the Genome project. 

Chuck I don't want to know. 

Mike But, it's an important gene to have these days, because they don't need help in finding and 
using information, but people were starting to use Gopher, and they needed assistance. 
And, even with Veronica and stuff like that, that didn't happen. So, it was really a joy to 
have these new tools, but people were overwhelmed by them. So, what did we do? We 
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hired somebody, they gave us a little money from the Department of Ed, and we chained 
that person to a desk, in a back room. It was actually our room next to the copy machine, 
and the person started answering questions via e-mail on the Web. We believe, although, 
someone else would say, "We did it first in ’82!" But, we believe we were the first formal 
information service that did that. And why did we do that? Because people kept saying 
that we need to add intelligence to the Internet. And artificial intelligence just didn't cut 
it, right? So we needed to use, not the next best thing, but an even better thing, and we 
called it “natural intelligence,” “people intelligence.” And I think, in a way, we are still 
doing that because the Web just makes it even worse, right? The stat I like to show is 
that the number of Web pages doubles every 50 days. So, you know, imagine, you know, 
Dave was talking about Alta Vista. Imagine if Yahoo really worked. Suppose you get to 
the screen, you type in two things, it automatically knows who you are, it comes back, 
and it has exactly what people want, that answers their questions intelligently, in the form 
they wanted, or whatever. I mean, we'd be all out of work. 

Chuck But, the truth is, they are terribly bad, and that's great news for us. 

Mike But what do we do with it? So what do you think? I know there are questions about 

where we're going and stuff like that, but it seems to be that there is still this tremendous 
need in spite of the Internet, the Web, all the search engines, and everything else (for 
librarians to help locate and evaluate relevant information). 

Chuck Well, you know, in the good old days,. . ..that's what you were talking about, right? 
The good old days, were great because, remember that phrase, "You could fall into a 
gopher hole?" Well, I think we can still fall into the digital reference hole, as well. 
Because part of the problem, and I think this is what we really need to get down to, Mike. 
I mean, you and Dave are sitting here [going], ‘Rah, rah, rah!’ Let's get at how far we 
have really pushed the envelope in the digital reference environment. I really think, that's 
where we need to begin. And the fact of the matter is that there are plenty of places out 
there in the Web environment where I don't need a librarian. 

Mike OK, fair enough. 

Chuck It's unclear to me, and I think it's unclear to a lot of people, whether you are going 
to get the same, different, better, worse —what kind of quality [service] do you get in one 
environment vs. another? I don't think we know. I'm not sure what evidence there is out 
there. I think we're going to look at a couple of the commercial sites later on and think 
they’re great! I love it: Dave commented that when you go up and talk to a reference 
librarian, they can see that you're in a hurry, they can see [you], they can make eye 
contact. My grandma was great. It was great conversation, 'cause here's how my 
grandma talks to people she goes, (shows a lot of facial and hand gestures), and she never 
said a word! I was trying to do this the other day with my digital reference service, 
providers, and you can't do this (gestures). We need some interactive video. How long 
has CUSeeMe been out? 

Mike Actually about 10 years, Cornell University. 

Chuck Yes, and here we are doing e-mail reference. 

Mike Yes, but let's give us a break. It's still the beginning, we're still moving through things. 
But, you can give clues in e-mail reference as well. In fact, I think there are some digital 
librarians that would tell you that, when they have been interacting with clients through 



0 



the text, and some of those contacts are not getting in the way, that people are more 
willing to give you [more precise cues as to] what they are actually looking for. 

Chuck So a key issue here is, it's not only training us, with our net side manner, but we are 
going to have to help patrons and users. 

Mike I'm glad you didn't say train users. You said “help.” 

Chuck Help. 

Mike Help. 

Chuck [Help] them to understand how you give cues in this environment. 

Mike That's correct. 

Chuck It's a very different world. But, remember now, Mike, this is Chuck of the "55% 
rule." 

Mike I knew he was going to bring this up. 

Chuck In my younger days, “d-a-z-e,” by the way. And, for those of you who don't 

remember it, we did a lot of work, and the fact of the matter was, whether you were in a 
public library setting, an academic library setting, [or] a law library, on average, 
reference librarians gave [a] 55% correct answer fill rate, on quick fact and bibliographic 
questions. 

Mike And, that study has never been challenged. 

Chuck Other people have done it. I mean it varies; it could go up to 57% or whatever. 
Mike In Seattle, all questions are 60% and above. 

Chuck Right. I've got to tell you this: I was doing this presentation at ALA, and I was 
saying it's 55%, and this woman stands up and says, "Dr. McClure, you need to come to 
my library, because at my library we're at 85-90% correct answer fill rate." And, this was 
one of the libraries we had in the study! (Laughter) So, my point here is, inquiring 
minds want to know: What's the correct answer fill rate in the digital environment? In 
the Web environment, as opposed to traditional? Dave is right on the money, and I hate 
like hell to agree with Dave in public, ok, but we don't [know] what [the] cost per 
reference query is. I'm a director; convince me. Convince me that this is something that 
I should spend my money on, because, excuse me very much, my budget is tight. Now, 
I'm not against it. I'm just saying, where are the data? Where do we go? What do we not 
know about these things? I think we are starting to get at some of these quality issues, 
and I hope we can get back to this, 'cause I'd really like to get emotionally upset about 
this. (Laughter) 

But, before we get to the quality stuff — I know that's there, that's one of our bullets, and 
he's going to try to hammer home on this stuff, but I have a couple of rejoinders. What I 
really want to talk about first is that when we talk about digital reference, and you talk 
about the environment that we're going to be dealing with, it's not just today, it's not just 
asynchronous-based, whether it's Web or e-mail. Do you remember the other day, we 
took Chuck and his wife on the lake tour of Lake Washington, and we wanted to go to a 
restaurant afterwards. We didn't remember the name of the restaurant over in Bellevue, 
but we did remember the type; it’s kind of a tapas restaurant, you know, Spanish. First of 
all I tried my Web browser, and I do have a Web browser on this (waves cell phone), and 
I do check ball scores. I was doing that all afternoon. But we quickly called SPRINT 
PCS, 411, and do you realize that when you call now, they don't say, "I'm sorry, I don't 
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have that name," they say, "Well, what kind of restaurant are you looking for?" And, 
they actually engaged in a question-and-answer with me, and they wouldn't let go. And 
then after they finally got the number, she said, "Would you like me to call and make a 
reservation for you?" She didn't say, "Would you like me to connect you?" 

Chuck That's great stuff! Why don't we do this? 

Mike That's what I'm saying, give us a chance to get there. 

Chuck We only have 20 years to retirement, man! 

Mike Anyway, so. . . . 

Chuck You know the part that he didn't tell you about this though was, that when we 

finally got to the restaurant... you have to love Seattle, and this thing with coffee. So I sit 
down and I'm listening to people ordering coffee, and it's like a strange tongue. "I'll have 
a double hit, very thin, no fat, espresso..." (laughter) 

Mike That's why we need digital reference, to help out at counters. To know how to order 

coffee properly in Seattle and things like that. The other thing I want to point out is the 
commercial side. I don't know if I'll do it right now, maybe we should move on, and I 
may come back. How many of you have checked out the various commercial sites 
lately? How many of you have checked, not just Ask Jeeves, but the Ask Jeeves Answer 
Point? How many have looked at that? How many of you have looked at [the] 
LookSmart site, recently? Hands. . ..hands. How many have looked at InfoRocket.com? 
HowaboutEXP.com? See, a little less there. AndthenthereisAbout.com. I’ll see what 
we have on these, and, maybe I'll show a couple. 

Chuck This can be a real interesting threat to what we're doing here. This is a complicated 
environment out there, and there is competition, serious competition, for what we do, and 
how we do it. 

Mike But, I think that's a good sign, because, remember the first battle? We talked about 

access; there is the access battle, right? There are library catalogs, there is the front end 
to information databases and things like that, and then came the Web, right? We all knew 
when the Web first came out, and we had these wonderful things called browsers. But, 
that was not the right way to access the Web. We needed to do it another way. We didn't 
need a browser; we needed a searcher. And, we all knew that. Us index-type people with 
the genetic defect, we knew that. But we didn't invent the damn thing, we let these two 
hotshots from Stanford invent Yahoo!, and they are the multi-billionaires, instead of 
librarians today. And that's why when I see stuff like this, when I see InfoRocket, or 
something like that, when I see LookSmart Live, and I see they have a list of people with 
charges [for fees] and things like that on them. I don't know, start Navigator, it's ok 
(gestures towards laptop as Web browser launches). I'm an IE guy, I live in Seattle. It's a 
requirement; otherwise they take away your stuff. But, you start to see some of these 

things that these guys have here on these sites Here's the InfoRocket site, and you can 

go into the Ask A Question [text box], but they list people, they list other things on here. 
You've got all kinds of stuff. In the EXP.com site, there's the expert way to ask a 
question. This is not a live connection. That's why we're having a problem here. So you 
get to see all the different things, and then after you connect to one of these sites, it has a 
list of experts... 

Chuck Well, doesn't it also let you make a deal with how much [to pay for an answer]? 
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Mike Yes, and how they want to charge, and make a deal. Wouldn't that be nice if we did that 
at a public library? You could either get someone that doesn't have an MLS from my 
school, but has one from your school, and then you . . ..(that was a joke, folks) you pay 
different amounts, depending on their expertise. And it excites me that these places are 
now doing this kind of thing. Because I think that it speaks well that the world is coming 
around [in recognizing the importance of reference work]. This is an important 
development that we need to think about, and what we have today, which is e-mail-based, 
flat reference, [with] no signals, is just the beginning. 

Chuck So you think that library-based digital reference can hold its own in the 
marketplace, Mike? 

Mike Not only do I think that it can hold its own, I think that in a free society, library-based 
digital reference is if it is not the future, then we are in trouble as a society. 

Chuck Now that is something I can agree with. The fact of the matter is, if we lose this 
opportunity, right now, this moment, this second, to make the digital environment work 
better than how it's worked in the traditional environment, I think we're in bad shape. So, 
Dave's call to arms of, "we need to do stuff now" is right on the money. 

Mike The question is: What do we need to do? Where do we need to go, what are some of the 
issues? Issue #1 [is] complacency. How many of you are doing digital reference out 
there? (Show of hands in audience.) It's not good enough! Because you are complacent. 
You're not doing synchronous [reference transactions], you're not doing wireless, your 
[reference services are] not imbedded in devices throughout society (waves cell phone). 
You're not competing directly with intelligent search engines and things like that, or 
agreeing] to deliver [these services]. The customization: every time a user comes to you, 
it's a new interaction, right? 

Chuck Here's the story, y'all: “Oh, for my library, we do a pretty good job with the 
resources we get. We do ok, for a library our size.” Ever heard that one before? 

Baloney! I don't want to hear about “good enough service!” 

Mike Right. 

Chuck What I want to hear about is outstanding, unbelievable, exemplary service that 
BANG, knocks everyone dead. “Oh. . ..we do pretty good for the budget we get.” OK? 
That's it. I've had it with that. 

Mike Because in the library world, across libraries, we're talking about school, public, private, 
academic, special, whatever, we have thought that “good enough” was good enough. 

Chuck “Good enough” is not good enough. It's the whining thing “we have a director 

that doesn't like. . ..oh, we don't have enough money.” It's unbelievable. OK, there it is. 
Give me a break. 



KEY ISSUES 

Mike So, what are the issues? We made our point. I think we agree on something. 

Chuck What do we agree on? 

Mike We agree that this could be important. Libraries could play a central role [in providing 
networked information to the public]. That this is vital in order to have the Internet used 
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in an effective way to meet people's needs. All people. So, where do we go from here? 
We buy in. We agree, Chuck, that “good enough” is not good enough. 

Chuck OK. 

Mike What are the issues? 

Chuck I'm going to bullet out some of them, and I'm going to try to talk about some of 
them. And we'll let Mike [join] in every now and again. But let me bullet out a couple 
[of points] and just talk about what they are, and we'll come back, ok? We can come 
back and go into more detail. Issue #1: management. It's unclear to me whether we need 
to re-invent the management organizational structures which we're currently using for 
digital reference. It makes me absolutely nuts to think that 9,000 libraries around the 
country are trying to do the same damn thing with the same damn sources, and let's just 
re-invent everything for every library in the country. Give me a break. 

Mike Management - gotcha. 

Chuck Second one. How about connectivity issues? I know, I know, no one wants to 
hear about the digital divide. We just finished a study for a National Commission on 
Libraries and Information Science; and the fact of the matter is that over 50% of all 
public libraries don't even have 56k [connectivity to the Internet]. Now let me explain to 
you the quality service. . . 

Mike What percent of libraries have some connectivity? 

Chuck The good news is that 95% of the libraries are connected. The bad news is, let me 
tell you what you can do with 28.8 [connectivity]. OK? And forget T-l [lines]. We 
should be talking, what is it — OC3? OC5, OC10-12? 

Mike Connectivity is still an issue. 

Chuck How about, there are still some training issues, there are still some help issues, 
there is still some learning how to work in this environment. 

Mike How many of you would say that less than 50% of your staff could really do quality 
digital reference service right now? (Show of hands in audience.) That's pretty rare. 

Chuck Yeah, but, you know, it makes me nuts. You want to be a digital reference 

librarian? Yes. Shazam! You are! It's great, I love it! And, excuse me very much, 
we're talking about these two damn good library schools, where you do learn this stuff. 

Mike What worries me more are some of these library schools. 

Chuck Don't go there, we'll get upset. 

Mike Things like this, [like] the AskERIC digital training that people go through, which has 
been expanded, the kind of [training] stuff that Internet Public Library does. 

Chuck It's great stuff. What I'm tired of is, “Shazam! You are a digital reference 
librarian.” We can do better. 

Mike OK, training. 

Chuck Let me talk about evaluation. I do want to put up a slide on this. Here's my 

favorite slide on evaluation. Everything I know, I know from Calvin & Hobbes. Right? 

Mike Right. 

Chuck (Reads from cartoon appears on screen.) (http://calvinzone.50megs.com/cgi- 
bin/i/ignorance.gif) 
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(B. Watterson, Calvin and Hobbes, 17 May 1992, distributed by Universal Press 
Syndicate) 

"It's true, Hobbs, Ignorance IS Bliss! Once you know things, you start seeing problems 
everywhere... And once you see problems, you feel like you ought to try to fix them... 
And fixing problems always seems to require personal change... And change means 
doing things that aren't fun! I say phooey to that. But if you're willfully stupid, you don't 
know any better, so you can keep doing whatever you like! The secret to happiness is 
short-term, stupid, self-interest. We're heading for that cliff! I don't want to know about 
it! Waaugghhh! (Falling off cliff) "I'm not sure I can stand so much bliss. Careful, we 
don't want to learn anything from this." (Laughter.) Now, my point. 

Mike Yes, what is the point? 

Chuck I'm afraid that we haven't learned from what we’ve done. There is a whole bunch 
of needed research; there is a whole bunch of evaluation questions that really need some 
attention here. What have we learned from how we do it? Yes, and Dave, second time 
today, you are right. It goes from best practices-kinds of evaluations, right up to various 
serious, funded research. How does digital reference work? How can we learn? How 
can we do it better? And that's what you were talking about earlier, right? 

Mike No question about that. Keep going. You were supposed to finish the bullets five 
minutes ago. I want to see which ones I choose to engage you in or not. 

Dave You know, Dave, you're not paying me enough to do this. 

Mike Is he paying you? (Laughter.) 

Chuck OK, then there are a whole bunch of finance questions. This is [a] “how much 
bang for the buck” question. What am I getting, as a director, out of digital reference 
services? What good is this? Does it really solve the questions? Are they answered 
correctly? What is the cost? And, because most of us live in what I call the “stagnant pie 
world,” the financial pie isn’t getting bigger, so anything in the library that is going to 
[be] fund[ed] differently] or [funding] new stuff, means you take it away from someone 
else in a library. 

Mike No. Wrong. I couldn't disagree more. That's the problem with our field. We narrow our 
focus, and we think small. If we provide central services that are not just meaningful for 
library users who currently use the library — but if we believe that we can provide service 
and information and a valuable thing to the entire society, then the solution is not to live 
within your budget. The solution is to find ways to increase your budget. That's what 
you have to do. 

Chuck I agree. 

Mike And this city is the perfect example. The Seattle Public Library, the King County Public 
Library down in Pierce County, Tacoma, it can happen. 

Chuck It can happen, but it doesn't happen with complacency, and it doesn't happen with 
"good enough." 

Mike OK. 

Chuck Are we ok? 

Mike That's correct. 

Chuck OK. 
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Mike And we may need to find some financial models where we benefit from the commercial 
aspects too. For example, what if a public library is providing help desk, or digital 
reference services, to some of the local businesses in town, and meeting their customer 
needs through that, and there is a financial return for that, 'cause they know how to do it 
better, and some of those people are working in the public library, and they are able to 
provide services in that environment as well? 

Chuck So, we agree. One of the issues here is the well-known “ROI” concern: “Return 
on Investment.” 

Mike Right. And new models, not just... 

Chuck What does digital reference do for my local economy? For this, that, and another 
thing. Because we have to go out and tell our story. 

Mike That's exactly right. We haven't done that. We have focused sometimes on our narrow 
constituents and we have done that to an extent, because those are the ones that make 
immediate demands on us, just like Calvin. But what we need to do in our academic 
libraries, and [in] our special and school, and certainly our public libraries, is [to] think 
about our broader audience [and] who we are responsible for. And how we can meet 
those needs, and how important it is. If you think about all of the human health services 
in our public institutions, that potentially is our scope of who we can deal with. 

Chuck Just as an aside, Mike, one of the studies we're finishing at the Information 

Institute at Florida State is that economic justification, if you will, of what the economic 
benefit of public libraries is in the state of Florida. 

Mike Where can I get the URL on that? 

Chuck I'm not telling you. 

Mike Is it done? No, I'm serious. 

Chuck No, the preliminary slides are done; they are on our Institute's home page (at 
http://www.ii.fsu.edu/Proiects/St-Lib-FL/index.htmlj . And the short story here is that 
there are huge economic benefits to both the local community and the larger state that 
result from a range of library services. One of the most important being — guess what — 
digital reference, and the move to digital reference. So, your comment about how 
important it is for us to make the case for what we dt is important, to people other than 
ourselves.... 

Mike Other than ourselves, and the traditional people that use our library. I know the next one; 
I have the next one outlined too. The next one you are going to mention is the digital 
divide. The digital divide folks are not saying, “Isn't it too bad that we don't have access 
to digital reference services from our local public library?” That's not what they are 
saying, because they can't even think about that or know about that, but that doesn't mean 
that they have any less of a need. 

Chuck There's the key. 

Mike That's right. And so we need to be talking beyond the normal people that use our [library 
services]. 

Chuck Well, an interesting thing to take a look at is- ya'll are probably familiar with the 
NTIA, National Telecommunications Information Administration. They just listed their, 

I believe it's called, TOPS Awards for this year. And if you look at where $30 million 
dollars went, a lot of them went to what we would call community awareness, community 
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access programs, that did not include libraries. Somebody out there thinks that many of 
the keys for improved equal access and network environment include a whole range of 
organizations that don't begin with the L word. 

Mike One of things that digital reference can do, and I'm not talking about asynchronous, five- 
day digital reference, I'm talking about immediate information reference services through 
synchronous communication. Community centers, kiosks, and public markets, and what- 
have-you become library services. 

Chuck OK, so what you just said was we need to re-think what the context is for the 

provision of information services. And in a digital environment, my nutso friend over 
here, out on the cruise out on Lake Washington, he's sitting there, finding out the sports 
score [on his cell phone], doing this, doing that, you know he can't carry on a 
conversation, but he can work his phone. That's the real world out there now, despite that 
it's Mike doing it. 

Mike Well I saw that commercial for that IBM kind of thing, you know with the vision thing, 
and I've got to get one — I have to sign up immediately for that. But, it's not just me, it 
really isn't. In the Seattle area, we're talking about new relations between the public 
libraries, school libraries and the university library. We're not just talking about resource 
sharing. 

Chuck OK, let's just get to it. This is baloney. The public libraries do this. The academic 
librarians do this. The special librarians talk only to God, give me a break. 

Mike Well, they are special. 

Chuck And, they are special; I understand that. My point here is models of resource 
sharing, Mike, and we've talked about this. We got to get over this stuff! Oh, that the 
public people [are over here], oh there are the academic people,... we have to get over 
that one. This boat is one we're all in together. 

Mike And, it's not just sharing resources; we're talking about sharing services and going 

beyond the resource side. And, it is not just ILL that takes two weeks. With the kinds of 
things we're talking about in the digital reference world — that really gets down right to it. 

Chuck So, there's the bullet. And the last bullet I'll throw out in front of you is 

information policy. Those of you that know me a little bit know that I fight the policy 
wars in DC, and it's brutal there. And, we have some interesting policy issues that we 
have to start dealing with. Not the least of which is privacy, security, encryption, 
filtering, and The Freedom of Information Act. When do you divulge access? Part of me, 
when we talk about these policy issues, gets me back to the Calvin and Hobbes thing. It's 
like the guy who jumped out of the 45 th floor of the building, and as he's falling to the 
ground, someone on the 20 th floor says, "Hey, how's it going?" and he says, "Everything 
is fine so farrrrr r rrrrr." That's where we are, because these policy issues, and the filtering 
and encryption stuff, the security stuff, we haven't paid enough attention to these things. 

Mike I agree with that. I really do. But, what I'm concerned about is that it doesn't tie us in 
knots from going ahead and trying new stuff. I think that is really important. This 
management thing, for example, which you talked about — there is no question that if we 
are going to have true collaboration across types of environments and libraries, then we 
need new styles of management. This Library of Congress project, and I don't know 
exactly the details of it, and the kind of things that the VRD’s AskA Consortium is doing 
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here and these various groups, and OCLC is getting involved — this is good, and we don't 
want to have to wait. As long as it's not a linear thing, let's deal with policy issues, let 
talk about those, but let's not be afraid to experiment. Maybe Napster is not the answer in 
that environment, but Napster is helping [to] take a new looks at things and it's meeting 
people's needs today. The point is to figure out ways that we can share resources and 
music and things like that that benefit everybody, and not wait until Congress passes a 
new copyright kind of thing, so we can't wait. 

Chuck If we wait for Congress to figure this stuff out, we'll be in the grave. 

Mike Right. 

Chuck So, Chuck says, at the local, state level, let's start solving these [problems]. Let's 
start [by] saying, “our policy, our position, is in the state of Washington, this is what we 
believe; this is what we are going to do, boom, boom, boom.” Because there is nothing 
else out there, or because it's messy out there, we better have [policies] that we 
understand ourselves for our own libraries. Otherwise, I believe we're going to leave 
ourselves wide open, and to quote my undergraduates: this is going to get "more worser." 

Mike I don't disagree with that, but I don't want to lose sight of the central thing. Maybe the 
policy issues within our schools and our field — let some folks work on that, and that is 
some of the people in this room. I'd rather have this group here focusing more on the 
issues of how we work together in a management sense, and the training thing. You 
talked about that. I couldn't agree more. We need to find new ways [to collaborate], and 
what we need to do is if everyone in their institutions gets some ideas as to the nature of 
the training that is needed, and the new models for training, using the technology for 
training as well within providing service, I think that we would be a lot further along. I 
really do think that again I agree with the policy stuff, but I don't want to get hung up on 
it. 

Chuck OK so let me be clear. I'm not against digital reference; this is great stuff. We're 
doing great stuff. But there are a number of these issues that we just bulleted [and] that I 
think we need to spend a little time on. Now, later on, I think we're going to want to 
come back and talk about mechanisms to get that done. 



QUALITY ISSUES 

Mike Right. We'll try to conclude with some of those things. But the bottom line is that the 
Internet is just still too damn dumb to leave it by itself. We can't leave people alone on 
the Internet because it's going to get worse and worse. 

Chuck NO! It's going to get “more worser.” 

Mike More worser, ok. That's why we need to put people into that mix. I want to come back 
to the issue you mentioned, the quality issue. And I know absolutely nothing about this. 
First of all I'm going to ask Lankes to come up. Where is he? 

Chuck He left. 

Mike How about Joe Janes. Joe, come on up here a minute. Does everybody know Joe Janes? 
From the Internet Public Library. What I would like Joe to talk about for a few minutes 
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is what do we know about quality, and what can we say. . .1 mean you have been in the 
digital reference business for what, eight years? 

Joe Five or six. 

Mike What was IPL? What was the first year? 

Joe March of '95. 

Chuck How do you know you are doing good stuff? 

Joe Well, that's a good question. 

Chuck No, answer the question! 

Joe The studies that we've done are not frighteningly different from 55%, to tell you the 
honest truth. The one that you and I did where we just sent out reference questions to 
people and had them use the Web, and not traditional resources, we had 63% correct 
answers. Tomorrow afternoon, mark your programs, there is a session on commercial 
and non-commercial Ask An Expert services. Two of my students are presenting, 
Chrystie Hill and Alex Rolf. We are going to find out -- I'm not going to steal their 
thunder — but there is a number in there that you're going to find very interesting. 
Especially in comparison to 55%, about how many of the answers to questions that we 
sent to commercial and noncommercial Ask An Expert services we could verify the 
answers to. And, we also ask things like, do they clarify. . . 

Mike I know that some people will be there, but a lot of us won't. You've got to give us a little 
of the answer. 

Joe Uhhh, No. I refuse to sell out my students. Go to the session. It's in double digits, I'll 
give you that much. 

Chuck Let's just get back to the quality issue. Do they — do we — provide high quality 
digital reference service, and how do we define high quality? What constitutes quality? 
If you don't know what the quality standard is, you're not going to be able to say you ever 
met the standard. 

Joe Well, that's exactly right. The standards issue. I think you are absolutely right. You 

listed a few. There is accuracy and verifiability, but there is also answering the question 
that was asked, which begs the whole question of the reference interview, and what you 
are and are not able to do in this kind of environment. There is also time to answer; there 
is also the affective component to it. Do people feel comfortable with the service? Will 
they return? Return rate, and willingness to return. Do they like it? My contention on 
evaluation is, first of all, those of you who are doing reference in the real world, on the 
desk, on the phone: How many of you evaluate that reference service on an ongoing, 
systematic basis? (Show of hands by audience.) That's exactly what I thought. Number 
one, shame on us all. Because, A, we don't do it. I think there is a lot we don't want to 
know. We don't know how much it costs, we don't know how good it is, we don't know 
whether people like it or not, we don't know whether we are doing any good, we don't 
know even if we are answering the right question. I think we should be ashamed of 
ourselves as a profession for doing that. 

Chuck Here's my question to you. Let me just propose a couple of quality standards. 
Here's a quality standard that 65% of the digital reference questions will be answered 
correctly within a 6 hour period of receipt. There is a quality standard. 

Joe I'd like it to be higher than that. But 65% is ok. Within 6 hours. 
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Mike Can we do that? How many are involved in digital reference? Raise your hands. 

They're afraid now. That's what I thought too. 

Chuck Because this is what I also call a Ouiji board evaluation. Ouiji board evaluation is 
“will you provide a 65% correct answer fill rate within the 6-hour time period?” 
(Pantomimes using a Ouiji board.) MmmMmmmMmmm.. .Yes! Here's another one for 
you. 

Joe OK. 

Chuck 90% of the users of a digital reference service will assess the courtesy of digital 
reference librarians to be at least 8.5 on a scale of 1-10, 10 being high. Boom. 

Joe Good. I like that. 

Chuck Trust me, I do this for a living, y'all. We can come up with these. And the short 
story is, we don't. 

Joe Well, no but I think it would be a lot easier to know in the digital environment than it is 
in the real environment for two reasons. First of all, I think that when you are doing 
reference on the phone or in person, it's awkward to evaluate right after you're done. So, 
you're finished with people, and they are backing away, 'cause they are off doing their 
thing, and you try to push a questionnaire on them and say, "Oh, how'd we do?" which 
they don't want to hear. In the digital environment, you can send them a questionnaire 
over e-mail two days later. They've had a chance to think about it, internalize what 
you've done, and to reflect on the answer. And then you send the questionnaire. .. 

Mike OK, ok, you've cut into our time. . . 

Joe You called me up here! 

Mike Thank you very much, Joe Janes. Lankes. Come up here. There have been some studies 
on AskERIC recently about quality, right? Tell us about the AskERIC quality. 

Dave Damn good. What else? 

Chuck Well, right. 

Mike Well, wait. Damn good, or do you have some real stats? 

Dave There are real statistics. Makiko Miwa, who is actually here and will be doing a session, 
found that [there was] about [a] 90% satisfaction rate and positive response from people 
using the AskERIC service. 

Chuck OK. Stop. I love this stuff. The satisfaction baloney. 

Mike Didn't you do a major study on that? You got like $800,000? 

Chuck No. No. Here's what I love. A patron comes into the library. They can't even find 
the damn reference desk. They finally figure out where it is, they go up to the reference 
desk, and the reference librarian is sitting there with their head down, can't get eye 
contact. Finally the reference librarian says, "Yeah, what the hell do you want?" Then, 
the patron says, "Well, could I please use", and the reference librarian says, "Yes, it's 
over there by the National Union Catalog, over there!" So, then you give them an exit 
interview. They leave the library, and the librarians ask them “To what degree are you 
satisfied with the service that you received today?” "It was great." Boom. You think I'm 
kidding you? That's what studies show. It's like we could be sitting at a moat with 
machine guns. Satisfaction is so multidimensional, it's lost its meaning. 



Mike Dave, since he hasn't given you the chance, could you go into a little more detail into the 
dimensions of satisfaction which was covered on the study? Don't disappoint me here, 
Dave. 

Dave Yes sir. Well, that study was conducted based on Bob Taylor's seven aspects of 

evaluation. It went into things like timeliness, accuracy, filtering process, and it really 
had a rather in-depth set of criteria. What was interesting was that Taylor's aspects. . . 

Mike Value-added model. 

Dave Taylor value added model was developed for systems, not specifically for digital 

reference. But, with a little more work, they added three more dimensions to it. And it 
works extraordinarily well. 

Mike So, there are some more specific dimensions, which is exactly what you are talking 
about? 

Chuck Absolutely. 

Mike We need to be much more specific then to say, “Are people satisfied?” Are they getting 
accurate information? timely information? Has it narrowed the information down to a 
manageable amount? Has it met their needs? 

Dave One of the interesting things that the Virtual Reference Desk has done [which] has 

developed out of a group called the AskA Consortium - [is] to develop a set of quality 
criteria. They call it [the] “Facets of Quality Document.” It was a very interesting paper. 
It was the first time we had an expert panel put together a series of characteristics and say 
“These are some facets [of a quality digital reference service] you should look at,” and 
then we added dimensionality, so level of performance and different benchmarks within 
[were incorporated]. They are slightly different when you talk about networking services 
together. For example, one of the first criteria was non-biased “we will be a non-biased 
service.” We were at a meeting, actually Joe was at the meeting, and brought up, "You 
know, in a networked environment, you don't necessarily want a non-biased environment. 
Sometimes the bias is what you want. You want a point of view, you want a context." 
And so, yes, I think there is good work happening in the Facets of Quality, and I think 
what's interesting is while a lot of it carries over, it's not a one-to-one match between 
traditional reference and the digital reference environment. So there is a lot we know, but 
there are some different twists on it, particularly when you begin to network the services 
together. 

Chuck But, let me be blunt. If you come away from this session, and you say, "Sheesh 
Chuck, man, he's nuts, we've got to do evaluations..." OK. But that is not defined as, 
"Were you satisfied with our service today?" That's not what I said. Satisfaction studies 
are only a very small part of evaluation. 

Mike Thank you Dave, appreciate it. All right, so we're starting to tie this up. We do want to 
leave some room for questions, and stuff. I want to summarize where we are and then 
move to a conclusion type area that looks ahead towards the future. First of all, we have 
said that quality counts. Quality is important, a multi-dimensional sense of quality. We 
have said that training is crucial across the types of questions that are answered. We said 
that this issue of new forms of management that in fact move institutions to work 
together in new ways is important. The other thing I heard you say is that “good enough” 
is not good enough. We need to be bold in this; we need to move really ahead and put 
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ourselves out there. That I second and really promote this, because I do feel that we are 
at a revolution. But, again, like the Web browser and searcher war, I think it can be very 
quickly that we lose this battle, and wind up being second fiddle to the commercial sites. 

Chuck Yes, I guess, I'm sort of with you on that. I'm less concerned about the AskA wars, 
if you will, between the commercial folks [and libraries]. And, it may be a good thing, 
Mike, you're right. It may be a good thing to get us to sharpen up our thinking on these 
approaches. 

Mike Well, when you talk about digital divide there are those who can afford to pay 50 cents, 
or a dollar or three dollars for every time they need a question [answered]. But the vast 
majority of people [cannot]. 

Chuck This won't be a problem in the future because [in] the next administration they are 
going to have Internet, digital ref, voucher systems. 

Mike That's right. 

Chuck You didn't know about that? 

Mike Well, in my latest meeting with George W. . . 

Chuck Never mind. So, what we are coming down to here is. . .we really do need to push 
the envelope. What we do is neat stuff; it's still evolving, but we've got an envelope to 
push. Those bulleted issues that we talked about are not going to go away in the short 
term. They need some attention, and one of the things I'd like to propose is a 
conscientious research agenda in this area. When I say a “conscientious research 
agenda,” I mean that where there are teams of people [who] are working on some of the 
issues we talked about, as opposed to, "Oh, did you hear what they are doing over 
at... wherever?" 

Mike But wouldn't it be great if we could have it. Maybe it's Library of Congress or IMLS 
[Institute of Museum and Library Services], or what have you, that provide a set of 
evaluation mechanisms that every digital reference service can implement. You talked 
about recreating wheels, that if each of these people has to go out and develop their own, 
that's not going to do it. 

Chuck Well, there is a model here. The model is, just as an example, anybody here from 
Michigan? Yes, some hands. In the state of Michigan, the public libraries in Michigan 
said, “Excuse me very much I'm tired of the way funding occurs here,” [so] 115 public 
libraries got together, kicked in "x" amount of money to get something done, and boom, 
they had a research project. I am of the opinion that if you have to wait for others, if you 
have to wait for these [grants], and all this other stuff, it is going to take forever. We 
need a clearinghouse to be able to exchange what the best practice information is. We 
need an agenda where we actually go after what these quality standards are, and how to 
demonstrate it. And, we need to do it now. This research needs to start now! 

Mike The clearinghouse exists. That's the AskA Consortium, and the VRD group here. And, 
in this state, actually, I'd have to say, in a positive way, the state library would be very 
receptive and ready for that. We are working together, and moving pretty quickly. What 
I'm worried about is that we're all going to go home, and we like all this stuff, and it’s 
great and it sounds good,. . .and we're going to start to implement some of these training 
programs, and doing more digital reference and that stuff. But, and I mean well, Chuck, I 
don't want you to yell at me next time I come, you know. I want you to say, "Mike, you 
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did a good job in our library." But, I don't have the expertise or the time to get out there 
and develop it [alone]. 

Chuck Chuck says we need to invent these evaluation methods and quality standards one 
time. 

Mike That's what I'm saying. 

Chuck You heard my colleague, Joe, say, “Yeah, but Chuck, here's some quality 
standards. Here's some criteria. Now what do we need to do?” We need to 
operationalize them; we need to proceduralize them, so that we're all collecting data in 
the same way. 

Mike So one of our recommendations to Dave, and [to] Joe, and the group in this room, is that 
we need quality standards that can be used by all of us in a simple, direct way, to measure 
the effectiveness and quality in digital reference service. 

Chuck So, then we stand up and we say, “Hey, look at what I do with my digital reference 
service.” My point is, we can't say that right now. 

Mike And, the second thing I would say is the training side. We need to be able to have 

established training [guidelines] that we can adopt, and then use and implement at a local 
library or information center. 

Chuck Help me with this. You are a library educator. Is that right? 

Mike No, I'm an administrator now. 

Chuck Here we are with a number of library educators in the room. I think we need some 
instructional modules that very clearly say, “Here are some skills. Here's how to get 
them related to digital reference service.” Now, maybe we're not quite sure what those 
are, and how they are doing it. Let's find out. The “Shazam” approach to being a digital 
reference librarian isn't where it's at. 

Mike This thing about — that you can just do it by doing it, I agree with that. I want to be 
careful. We haven't thrown out the term “librarian,” and it sounds like a very specific 
sense. There are people in the room that are doing digital reference who are not 
necessarily librarians or working in libraries, and that kind of thing. We're broad enough 
to that. We're using that as a model in the example, but in essence, Ask Dr. Math, and all 
the other AskA services, they are part of the same community. But, I do like the idea of 
somehow certifying the "Good Housekeeping Seal of Approval," the "Eisenburg - 
McClure," certification of digital librarians. 

Chuck Let's talk about this just a second. The problem here is that anyone can say “I'm a 
digital reference librarian,” [and] then you have to wonder where the quality is. Now, 
this is scary stuff, isn't it? This is kind of an unknown area, here. But one of the things 
that we hear all the time is, “How do you know [when you’re] getting good service?” 
Some of you say, “Well, I know if I go to this Web site, it's good stuff.” How do we 
know when you're going to digital reference librarians, that you are getting good stuff? I 
think we need to think about it. I'm not sure I have the answer right now. 

Mike If we have consortia, and if they are sponsored, if you are part of the VRD AskA 

Consortium, and you know that you are getting digital reference service through an AskA 
service, the burden can't be on the end user, either. It's got to be through our libraries or 
something like that, that say, “When we set up these mechanisms that our patrons are 
getting answers through whatever system, that we know that has a certain quality base.” 

O 
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Chuck OK, let's go to the next one. Why can't we do a better job of exploiting the 

technology that is out there, in the digital reference environment? Whether it's interactive 
video, or whatever, we are not doing it. 

Mike I do agree with that. What concerns me, and I know we have to set up these databases 
and work on the interaction of how we route things, I've got to tell you, the future of 
communication in the network environment, in the world, is not e-mail. 

Chuck No? 

Mike It really isn't. And, it's not voice mail, either. 

Chuck No? 

Mike I think some form of real-time digital reference, and we've [been] talking about it and 
doing it. How many of you are experimenting with synchronous real time reference, 
through the networks? I like that, share that. Put that out there; let's hear about it. 

Chuck What about real-time when you are in your library's Web site, and you can click on 
“Ask A Librarian.” 

Mike That's good, because a year from now it will be double or triple [the number of libraries 
offering real-time digital reference]. So, that's where I think we're going. I think we need 
to do it with various devices. I don't use a Palm [Pilot] anymore. I have my HP Jornada 
and I have my cell phone, [and] I want digital reference help through that. 

Chuck Did you ever see the movie, "Mr. Gadget?" 

Mike No. (Laughter.) 

Chuck Go ahead, sorry. 

Mike No, it's not just a gadget. I want to see.... here, look, look (waves cell phone). It says 
here right now under the Middle East Crisis, I'm reading, the news is the summit is 
working out, here Clinton is talking. He appealed for Mid-East peace, and whatever. 

But, I want some background on it, you know, my nephew is over there, and stuff like 
that. I want to find out more about it. What I'm saying is, real time [information] through 
various devices. All the technology, it's really there, and as networks and technology get 
more intelligent, and they will, then we incorporate that. We do meet user's needs, as 
much as possible through technological needs. As someone once said, "Any teacher that 
can be replaced by technology, should be." I'll say the same thing about librarians. Any 
librarian that can be replaced by technology should be. 

Chuck The rest of this that is going to be scary is. . .remember, a lot of these sites are 24/7. 

Mike 24/7. 

Chuck That's the world we're living in. And so when we say, “No, I'm sorry, we close the 
library at 5:30 today.... and if you please leave a note, we'll get back to you.” [The 
patron] wants to know right now. We're talking real-time, all time. Real-time is different 
than all time. That's the world we live in. 
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Mike To leave time for questions, I'll go first, and then I'll leave it to you. I think we need to 
look forward, and not backward. I think those of you that have not been involved with 
digital reference, you may not want to even worry about baseline e-mail reference and all 
kinds of involvement with that. You may want to jump right to some kind of live 
interactive or something. I don't know, I'm not recommending that purposely. We need 
to look together. We need to look at users. Who is the new user? How are users 
different today than they were? And, they are different. The Web is a reality that we 
need to know about. Not know about, but accept, and work it into what we do. Dave 
talked about context. And that's a good word: Context. Context. Context. And, figuring 
out the major questions, and the nature of users, and the nature of our services, and how 
that [all figures into providing effective digital reference service]. And I would add 
collaborate, collaborate, collaborate. And I don't just mean connections. I don't mean 
coordinate. I don't mean linking. We need to truly collaborate, because the end user 
doesn't care whether they are really getting library services from the University of 
Washington Library, or the Seattle Public Library, or the Michigan Library. They want 
service and information. Third thing [is to figure out] costing models and economic 
impact. We can't throw up our hands and say, “Well, there is no way to do that.” What I 
talked about before is not trying to live within our current means, but finding new means, 
new money, to do these kinds of things because it is important stuff. The last thing I'll 
say is the teaching and training role. That is because the reference question has to move 
from beyond even just getting an answer to folks, but this issue of helping people to 
become information problem solvers. And that doesn't mean that they become self- 
service and self-users. But it means we become information consultants, and helping 
people to better define their information needs, and to find ways to meet those needs. 
Now, with this interview and this interaction, but also with future ones. To me, if we can 
pull off some of these issues, we're somewhere. 

Chuck I'm right with you. The beauty of letting Mike summarize this is that he takes 
everything that I wanted to say. But I do have a couple of other things. I think that 
where we've come in this discussion is kind of full circle. The truth is, and Stuart... is 
Stuart [Sutton] still in here? Stuart and I were talking earlier. ... I'm not sure where we 
are on this wave. You know, the digital reference wave. Is the crest still building, are we 
on the top of the crest, are we crashing? Now those of you that are old, like me that Joe 
pointed out.... I was there when we had the online database-searching model. Oh my 
God, we could get on and do online database searching. Any of you remember that? 

You could walk into the library and have an interview, or fill out a form, and someone 
would do a search for you. And now, that is just part of the reference process. No big 
deal, right? But we want this field of digital reference to be so commonplace that it's the 
norm; it's not different. It's the norm of what we do. OK, so that's point one. Point two 
is this is the pillar of what we do in libraries and information services. The pillar of what 
we do is meet user information needs. If we forfeit that responsibility, if we can't do that 
well, what do we do? So, it's not just a matter of having service, having this in place; it's 
us as information professionals that say “Put me in, coach.” This is stuff I do, and I do it 
well. 



Mike And, I got to remind us, again. I think that we gave up; we lost, in one way, the access 
role. The people providing access, and the tools for access today are the AltaVistas of the 
world, the Yahoo !s of the world. These are the ones that are actually more responsible 
for providing access to information than libraries are. That's another speech at another 
conference, but I think we can recapture that. Because quality information is [the] 
library. And junk is junk. Therefore, I think that the commercial services are something 
we need to look at, learn from, co-opt, and do them one better. Because, you are 
absolutely right. This is the center of our role. Meeting people's information needs is 
what libraries and the information profession is all about. 

Chuck The conclusion is: vision, commitment, do it. Just get going on this. We've 

offered a couple very practical kinds of next steps that we think ought to be done, and 
I've got to tell you, Mike, if I look through the program here, today and tomorrow, 
unbelievably good stuff. I hope that this time next year, Dave, when we're down doing 
this conference in Atlanta, or wherever, that what we can also talk about is how we move 
forward from this, on some of these key issues. Thank you all very much. 



AUDIENCE QUESTIONS 

Mike We have about 10 or 15 minutes for questions. There are microphones all up and down 
[the center aisle]. Can you introduce yourself to the audience, please? 

Carol I'm Carol Hert from Syracuse University. Another threat that I see on the horizon is 

sticking with the notion that digital reference as stand alone services. Every Web site has 
AskA places on it, and if we continue to think about this thing, that we're a standalone 
[service], we're going to miss this huge marketplace. And I go to a lot of Web site 
development meetings and I really do hear them talk about the same issues that we talk 
about. But they don't have people like us informing that discussion. Web sites in general 
are a threat, if we continue to envision this as [being made up of] stand alone services. 

Mike Very good. Thank you, Carol. Any questions or comments? 

Q1 I was wondering. . .You seem to be making the assumption that we need to have 100 % 
accuracy, as... have you guys been parents? (Laughter.) My daughter hit teenage years, 
and I suddenly am never right. 

Chuck No, I don't make that assumption. And, yes, we're both parents. The reason I'm 
nuts is because I have a teenage daughter. What more can I say? What I am making the 
assumption of, we don't know what we do right now. You can't be better if you don't 
know what you're doing right now. 

Q1 Also, in the economic industry, economists agree that not one of them agrees with any 
other economist too. 

Mike A 100% accuracy is not necessarily a definitive answer that is the truth. It may be a 

range of ideas, that speak to an issue. But I think the other thing that Chuck mentioned 
was [to] set the standard. You decide what an effective answer is. Then, you should be 
meeting that 100%, whatever that might be. 

Q2 I was just wondering, when we start to shift resources in the direction of digital reference, 

and in providing digital services, what happens to the information poor? By that I mean 
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the people that don't have home computers, don't have Internet access. We can spend 
80% [of our budget] on digital services and the other 20% of our resources on providing 
services in the library. 

Chuck Well, I'll take a quick stab at it. First, you are exactly right. What we are in right 
now is a transition period. The argument is, to what degree will the information have- 
nots, however you wish to define that, [be adversely affected by digital information], 
where that group will continue to be fewer and fewer, and we don't have to worry about 
that? My immediate take is that we are going to be stuck in this transition environment 
for a number of years in the immediate future. What we have to worry about is those 
folks that don't have access, and those that have unbelievable access. So in terms of 
resource allocation, this is going to be very tough. 

Mike But the information-have nots may need digital reference services more than anybody. 
Because those people are not the people that are walking into your libraries, frankly. I 
think that by having digital reference service to community centers, to schools, to after- 
school programs, to daycare centers and senior centers, and whatever, we can better meet 
those people's needs. 50% of America right now is wired to the Internet from their 
homes. OK, and it will probably get to 80%, and then we'll have the other 20% to deal 
with. We can make more inviting library environments, and try to get these people to 
come in. But the truth is, we have to go out to where they are. And, that's community 
centers, that’s recreation centers. And, therefore, how do we provide [assistance to meet 
those users’ needs]? It's through digital reference. 

Catherine I'm Catherine Sheldon from Seattle Public Library. This is a comment about 

accuracy in answering reference questions. I'm a fanatic about telling patrons when I'm 
sure, and when I'm not sure [of an answer]. And, don't we all wish that our physicians 
would do that. To say I'm looking at this source now, to the best of my knowledge, the 
way that I'm interpreting this now, I think it's 98% [accurate]. But you all know, and 
someone already pointed out, that statistical questions are quite controversial and so we 
need to be sure to tell patrons that just because you think I'm God, because I'm a librarian, 
and I'm saying something authoritatively, does not mean that it is the one and only 
answer. It's also important that when we do not find an answer to something quickly, and 
someone says, “I only have a few minutes,” that we tell patrons the fact that I did not 
find an answer now does not mean there is no information out there. 

Mike Especially the point about letting people know. That's [where] guidelines [come in]. To 
me, someone that answers a question by saying, “Well, here's what we've got. I'm not 
really completely satisfied with this, and I think we need more.... maybe it's 60% right.” 
To me, that's a 100% right answer. 

Chuck Yes, I guess what I'm concerned about though, Mike, is the famous old, “I'm trying 
to find a citation, blah, blah, blah,” and you're at the reference desk, and they say, “Go 
look it up in the National Union Catalog. It's over there.” (Points to a distant location.) 
That's not an answer. And the same thing can happen in the reference environment, 
where you say, “It's at such-and-such Web site. Good luck!” 

Mike And coming back to people, as well. Digital reference to this point has been single 

interactions, and maybe it's this trade-off with a little bit of privacy. I'm willing to have 
them know who I am, and what I am, so that I can come back and picks threads up. 
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Cynthia Cynthia Teague, University of Minnesota. I'm curious about your position. You 
seem to be saying that synchronous reference has to replace e-mail reference. I think 
there are a lot of questions that would be better answered in real-time. But there are other 
questions where people want to drop it off, and get an answer. They don't want to sit 
there while you're checking these sources. 

Chuck You are absolutely right. And the short story here is: We don't know. This is a 

situational model. In what instances does what type of reference service provide the best 
response? Situational kinds of criteria that frankly, when Joe was up here, we don't know 
what they are. Now, you just proposed one, which is great. Let's go out and find out in 
what instances does that model work? Because maybe what we want to be able to do is 
offer multiple methods of digital reference based upon the situational context in which 
people need information. One size does not fit all. Right? Do you agree? 

Mike Yes. You probably noticed we tend to get a little emotional, and exaggerate just a little 
bit to make a point. The point is we have tended to ignore the synchronous [transactions] 
coming [along]. I like the idea of a hybrid kind of a synchronous front-end interview 
kind of thing, and then an e-mail response later. I'm a little concerned about just e-mail. 
But, if I got a profile on file with AskERIC, and I'm a teacher and they know who I am 
and whatever, and when a question comes through from me, and they know who I am and 
they've got all that background, it's on a specific thing, and I can pick it up later. That 
would be terrific. 

Chuck Your point is a great one. 

Andrea Andrea McGlinchey from the Department of State, and I'm wondering if you could 
please address privacy issues a little more? 

Chuck As you know, because you work in the Federal Government, the Clinton 

administration has proposed specific privacy guidelines, and specific privacy policies that 
should appear on all federal Web sites. In recent work that we have done at Florida State 
we’ve found a lot of the federal agencies haven't done that yet. The other problem is that 
the federal guidelines for privacy are only that, guidelines. There is still plenty of room 
to wiggle around in. By the way, in case you don't know this, federal agencies have been 
FOIA’ed (Freedom of Information Act), to get lists of who have read websites; it is a 
mess. This has made many federal agencies, basically, at the end of each day, remove 
tapes, anything that had contact with any individual, where individual IP addresses can be 
identified. I can talk more about this. But the problem here is that we haven't recognized 
the problem. There still are questions in non-government situations. Can you be 
subpoenaed to release the IP addresses of who's been talking to you on digital reference? 
These haven't come up yet in case law, but they will. So, the short answer is: You have 
to have your policy statements approved by your boards, your governing bodies, so that 
at least you have that to fall back upon. And there are some great models out there about 
what those policy statements should be. But in a government setting, it is a very 
complicated matter because of the FOIA stuff. 

Mike I wouldn't mind seeing us, as a group, really jump into this. I'm very concerned about 
individual privacy in library situations. At the same time, I want these services to know 
me and be able to customize and personalize, so that I'm not starting from scratch every 
time. And these are sometimes conflicting values. We need to find some mechanism to 
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deal with that, and maybe we need some law and statutes as well. When I give up certain 
information like this, it cannot be subpoenaed and made available, but [the government 
may be allowing that] to happen. 

Chuck Short term: we can't solve it; long term we need some statutes and guidelines. 

Art Hi, I'm Art Gunn from Atlanta. 

Mike Are you going to host this next year, Art? 

Art Yes, we look forward to that. 

Mike OK. 

Art I'm curious if you will share with the group what role the Association for Library and 
Information Science Education (ALISE) has taken in terms of training for digital 
librarianship, if any at all? And, what would you recommend that that group involve 
itself in? 

Chuck Well, there are a couple of answers, and the short story is that ALISE probably 
needs to pay more attention to this. 

Mike And they haven't done that. I can pretty definitively say that I have been to the last three 
or four ALISE conferences and NOTHING has been done, and you are absolutely right. 
ALISE needs to do something. The schools can't wait for ALISE necessarily. They 
formed some consortia on distance collaboration, and we need to do more of that. If that 
is what you are saying, you see this more of an agenda item that we have to put out there, 
you are absolutely right. Now, Joe Janes, we've been teaching digital reference courses - 
and Syracuse does — but the next thing is for us to teach them jointly online [and make 
them] available to everybody. 

Chuck And, the distance ed. model in this time, right now, excuse me very much. ISU’s 
whole MLS program is available on the Web. SU's is, Mike's is about to be. I think the 
issue, Mike, is that we need some instructional modules that we can share and begin to 
work with this. Because if we wait for the organizations [to take the lead], I think it will 
be forever. 

Lisa I'm Lisa Roberts from the University of North Carolina at Greensboro, and I basically got 
up to say Hallelujah. I'm really ready to do this, and being from North Carolina, I just 
started thinking, you know, let's collaborate, because in North Carolina we're already set 
up with the UNC system, but also NC Live, which is purchasing databases cooperatively, 
and putting them in not only academic libraries, but community colleges, public libraries, 
and now we've finally gotten into K-12. So that sounds to me like a perfect venue to get 
together to start talking about this. So, who else is here from North Carolina? Let's talk. 

Chuck I love it. 

Mike That is a great way to end it. Thank you very much. 

Chuck Thank y'all. 
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Jeeves is no longer 10 people, but has grown to 700 people. 

Melanie Gardner, National Agricultural Library 

Melanie A. Gardner is the coordinator of AgNIC at the National Agricultural 
Library (NAL). She coordinates the AgNIC Alliance partnership and maintains 
communication among partners. Prior to her present position, she was the social sciences 
librarian in NAL’s Rural Information Center. Before coming to NAL, she worked for the 
University of Maryland at two different campuses. She holds a B.S. in Education, and a 
Master of Library Science with a concentration in rare books and manuscripts/archives. 
Ms. Gardner taught school for seven years and has been a librarian for over 15 years; she 
has been part of the transition from traditional to electronic reference services. 

Joe Janes, University of Washington, Seattle, School of Information 

Joseph Janes is assistant professor at the Information School of the University of 
Washington. He is interested in reference, particularly in the use of technologies to 
mediate and assist, and the use of networked resources in reference. His research is on 
models of practice in digital reference. He teaches courses in reference, online searching, 
research methods and statistics, and on the use of Internet technologies in librarianship. 
Janes holds an M.L.S. and Ph.D. from Syracuse University. 

Diane Kresh, Library of Congress 

Diane Kresh is director for public service collections and director for preservation 
at the Library of Congress (LC). Her experience at LC has covered a wide spectrum of 
responsibilities, including copyright, collections maintenance, document delivery, 
reference, user training, photo duplication services, and conservation. Diane serves on the 
LC Internet Policy Committee, directs nearly all the library’s general and special 
collections, and is leading the effort to provide enhanced Internet services for public 
researchers. 

Rivkah Sass, Multnomah County Library 

Rivkah Sass is Reference and Information Services Coordinator at Multnomah 
County Library in Portland, Oregon. 
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Steve Yin, AskMe.com 

Steve Yin is Product Marketer for AskMe.com. Since joining AskMe.com in its 
early days in 1 999, Steve has participated in many different aspects of the start-up 
business including recruiting, content development, media buys, and product 
development. Prior to working with AskMe.com, he spent four years with MSI 
Consulting Group as a Manager of Consulting, where he helped high tech clients such as 
Microsoft, IBM, and BellSouth.net with sales and marketing strategies. Steve has a B.S. 
from the University of California at Berkeley’s Haas School of Business. 
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Web Contact Center Software: 

Tools for Doing Reference in an Online Environment 

Steve Coffman 

Library Systems and Services Inc. (LSSI) 

Presentation 

Introduction 

This presentation provides a brief history of e-service and presents Web contact 
software as a solution to problems encountered in e-mail reference. LSSI’s Virtual 
Reference Desk is demonstrated as an example of software designed specifically for 
libraries. Related network services and issues are also discussed. 
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Email Reference 
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si Initially all that was available 
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~ Anywhere from 30- 50% of the libraries now offer it 


"T 


st Has not been overwhelming success 






o Slow turnarounds — people want it now! 






o Difficult to do a good reference interview 
o Librarian is stuck answering the question rather than 
assisting the customer 
o Limited usage 
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st Clearly another solution was needed 
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» Luckily ... one was forthcoming 
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Basic Facts About 

Web Contact Center Software 

■-» ♦- H * * * * H * — 

« Designed for live customer service in eCommerce 
« Allows agents and customers to work together over the 
Web 

Based on call center model for efficient, high-volume 
call handling 

« More than 50 vendors already in the market including, 
eGain, Lucent, Cisco and Siemens 
« All spending millions on software development 
R Solves many of the problems of email reference 
« Since reference is really customer service for libraries, 
it may be just the solution we’ve all been looking for 








The Reference Front End 
A Key Component 
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~ A marketplace of information services 
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If We Do Move Reference to the 
Web 

Reference 
“Filters” 

driving — 
distance — 
parking — 
hours 



People with Questions 



long lines at desk — 
busy signals 



People Who Actual ty 
Get to Ask A 
Question 




Normally Stressed 
Librarian 



M 



We’re Going To Need All the 
Help We Can Get! 



People with Questions 




No Filters on the Web 

* No Driving 

* No Distance 

* No Parking 

* Open All Hours 

* No Lines 

* No Busy Signals 
Way Over- Stressed 
Librarian 




Improving Reference Efficiency 



Self-help 
ft Tiered staffing 
a Access to subject expertise 
* Centralizing staff using call center model 
oReal Call Centers 
o Virtual Call Centers 
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* Call Center Magazine 

ohttp://www. callcentermaqazine.com/index.html 
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Bernie Sloan and Tom Peters 








« Call Center Solutions 












o httoV/www. tmcnet.com/ccs 






Technical Support: Lori Bell 








a Call Center Directory (old but they have Erlang C 
calculator) 






Subscribe at: 
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Further Information 
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For further information contact 
Steve Coffman 

Product Development Manager 







LSSI 

800-638-8725 ext. 265 
stevec@Issi.com 
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Building a Virtual Reference Network 



Susan McGlamery 

Metropolitan Cooperative Library System 

Presentation 

Introduction 

This presentation places Web contact software in the context of a cooperative 
digital reference network of libraries in Southern California. McGlamery discusses 
components and characteristics of this network and demonstrates the process for 
question-answering. 
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Components of the 


Building a Virtual 




Network 


Reference Network 




■“ — ;r - ^ :.a. 






§SWeb contact center software 






ISTiered Reference 


Virtual Reference Desk 






Conference 




SIS Subject expertise 


October 2000 










SSCooperation between libraries 



Web Contact Center 
Software 



IS Communication: 

Elchat, e-mail, Voice Over IP (VOIP) 
IS Collaboration: 

IT-) send URLs, Form Share, Follow Me 
IS Referral: 

Olive or by e-mail 
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Authentication 




Live Chat 




Screen ehot used with permission from Santa Monica Public Library. 



Pages on Hold 




Screen ahol used with permission from Santa Monica Public Library. 
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Subject Specialists 




After-Hours Reference 


^Metropolitan Cooperative Library System 




seVirtual staff, can be located anywhere 


Reference Center 

$§Los Angeles Public Library FirstSource Database 
^Consumer Health 




^Answer questions from all participating 
libraries' patrons 

38Quick answer/Internet questions only 
3€Referral to originating library for all others 
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Tiered Service 



SSQuick Answer 

Ellibrary hours, location 
©opac or ILL help 
©ready reference 

3?Refer to originating library for all others 
©live reference 
©e-mail 



http://www.247ref.org 




36 For more information, contact 
36Susan McGlamery at 




36213-228-7568 

36smcglamery@mclsys.org 
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Cornell University’s LiveHelp Service 

Paul J. Constantine 
Cornell University 

Presentation 

Introduction 

Cornell University’s LiveHelp service, launched in January 2000, provides live, 
interactive reference service over the Internet using commercially developed e-commerce 
software. The presentation outlines the goals of LiveHelp and describes the process used 
to design and implement the service. Cornell’s experience using LivePerson is discussed 
and user and administrative features are demonstrated. 



^ Cornell l^nivepit\\ LiveHclp 
Service / \ 



Paul J. Constantine 
Cornell University 
pjc6@cornell.edu 



Soft \pafe Requirements 

• S^ports private chat yC j 

• ©oes not require special^software or J 

/ hardware for user / \ ' 

Logs or captures transactions \ 
f Supports both Internet Explorer arid 
f Netscape / 

• Supports both Mac & PC 
• Allows multiple simultaneous connections 
* User friendly at both ends 
• . Al l ows cutting and pasting from librarian en d 



^Our Goals J 

To explore new models of reference 
^service \ / \ 

To provide live, interactive, synchronous 
reference service via the Internet 
To begin to develop a reference service 
for the digital library 
To explore e-commerce, customer 
service software 



• Reasonable'startup and^on-going costs 

• Enables operator to ignore^or block 

/ troublesome users / \y 

Audio cue for incoming queries 
• Customizable "branding" of front end 
• Fast response time 
• Good tech support from vendor 
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fare Desiderat 



• Supports embedded urls\ j 

• /Supports sending in^gesN^ 

• f Receives initial query in "private" 

/ environment f \ 

• Customizable or adequate size query 
typing box 



ldenjtified Similar / 
r o j e ct S / 

Univ. of NorthTexas / \ j 
los Angeles CountWPublimLibrary FYI 
Service / \ 

Lands’ End / 

Temple U. {Talk Back & Talk Now} 
Lippincott Library at UPenn 



Software Desiderata 

/ \ W j 

• Floating chat window jC j 

• Shows us user’s domain/l^address 
•Allows us to operateAervice^rom 
/choice of locations/computers^ 

/• Incorporates FAQ in front end 



• Webmaster 

• iLivePerson 
•/ParaChat 






Evaluated 



Webline 

Review Source for chat clients: 
Downloader's Guide to Net Chat 
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Ability to survey user^f / 

Ability to e-mail targeted groups of 
users j \ 

Database of “canned" responses 
Call logs /’ 
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Staffed by both librarians^and 

L rA_ : A.,_ X . 



,paraprofessionals ir 



two hour 



/ bl° c k s J \ 

/ Scheduled as an additional service 

f # 

point / 

• Staff are usually at their desks; rarely at 
the reference desk 



j S^me Pros of 
Person 

Database of “canned answers” 
Affordable price\ 



Some Pros of 

s** \ / 

LivePerson 



^ ^ 

y/eb-based; no^special software required 

yExcellent administrative^features / 

f No computing overhead \ 

Attempted connections during down hours 

can generate an e-mail transaction 



ime Cons of 
JvePerson 

\ A . 



• ^s all transactions gc^through 

/ LivePerson sen/ers,/esponse 
be sluggish 7 \ 

I ftrp rrtki ipI pi irmArt fr\r Mntppo 



Less robust support for Netscape and 
Macs 

Company focus is on e-commerce, not 
libraries 
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X 



Some Cons of / 
ive Person j 

Lack of locaf'control qveNsome features 

/ \ / \ / 

(Inability of librarian and user to share a 

browser window / \ 



B e n epts^ofL i v e H e I p / 

j^^\S e rv ice j 

Dial-up users^do not have^to give up 
ftheir connection to call reference f 
As computers become ubiquitous in 
libraries, users can access reference 
help from anywhere they happen to be 



Benefits of LiveHelp / 

(^\ Servibe 7 

Remote users have access to live/ 
(interactive reference4ervice 
Constraints of place, if not of'time, are 
lessened / 

Reference service is now part of the 
digital library 



Benefits of LiveHelp 



Service 



• We can provide interactive service to 

/ Cornellians who are^truly remote (e.g. 
Cornell in Washington, Cornell in Paris, 
etc.) / \ 

• It is a start toward a reference service 
that is both “high tech and high touch” 



ERIC 
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Next Steps 

X,o 

/ \ 



^e-evaluate software^selection 
(Market more neavil^/ 

Expand hours to cpver thoseViost 
desired by users > 



ForJVliW'e^l nf o rm at i o 

• Paul Constantine - pj/Gt^cornell.e'du 
or 607-255-33 N 19 ' v 




/ 



www.librarv.cornell.edu/okuref 



www.liveperson.com 




Next Steps 

! X 



tjer libraries at 



[look at partnering with 

/ Cornell and outsidey2ornell^ 

Examine how to integrate desk 
reference, telephone reference, e-mail 
reference, and LiveHelp 
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The Use Of ICQ In Providing Real Time Reference Services 



Joseph Yue 

University Libraries, University of Colorado at Boulder 

Abstract 

While e-mail reference is one of the popular methods of providing assistance to remote 
users of library online resources, it does not allow a real-time reference interview. More and 
more Web retail companies have started providing real-time customer help lines, yet libraries do 
not always have the same financial resources to implement such technologies. The use of ICQ 
for reference and as office hour consultation in an academic library will be reported. Issues such 
as staffing, technical implementation, and marketing of services to specific user groups are also 
be addressed. 



Introduction 

This paper reports on an exploratory project that uses free chat software for real-time 
digital reference. The software is called ICQ from ICQ, Inc. and requires little technical or 
computer knowledge. In addition to a summary of the project, this paper will discuss different 
features of ICQ used to enhance digital reference interactions and issues related to implementing 
ICQ reference services. Possible uses of existing models and potential partnership outside of 
libraries will be briefly addressed. 



The Project 

It is important for librarians to pay close attention to the help-seeking behavior of users 
online. Reference services should be as accessible and convenient as possible. At the same time, 
it is not enough to assume the public knows of the value of reference service. Librarians need to 
aggressively promote what they can do, especially today when many people have the 
misconception that everything is on the Web and is free. Lipow describes the concept of digital 
reference in this way: “Rather than thinking of our users as remote, we should instead recognize 
that we are remote from our users” (1999, p. 51). 

The goal of this exploratory project was to determine how reference librarians could 
reach out to remote users and provide real-time reference assistance as close to face-to-face 
interactions as possible using existing resources. The project was also the first step before a 
possible department-wide pilot project. Specifically, this project aimed at exploring the use of 
the free chat software, ICQ. A small-scale project (single institution) has its limitations, but the 
issues and problems encountered were general and fundamental. Thus they would be applicable 
to other real-time digital reference projects within the context of a single institution with limited 
staff time and financial resources. Furthermore, little has been written about the use of chat 
software, especially ICQ, as part of library services. 1 



i 



Jan Zastrow (1999) outlines her use of ICQ for her information broker business. 
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Assumptions 



Users can be grouped into two different categories: general remote library users and the 
target population in this project. 

Before the popularity of remote access to online databases, library users had to be on site 
to use the resources. Reference desks were able to provide the assistance most users needed. 
However, as more and more resources are available on the Web (with or without access 
restrictions), reference desks have fallen short in fulfilling this role. Among library users in the 
Internet age, few come into the library to ask for help or call on the telephone. E-mail reference 
does not provide the synchronous service most users need nor the quality of service they are used 
to getting at the reference desk. Telephone reference is a possible solution, but few people have 
two phone lines or access to a network enabling them to call the library and use the Internet at 
the same time. In fact, at the author’s library, most callers who request search assistance are 
faculty who are in their office and have the luxury of simultaneous digital (Internet) and analog 
(telephone) communication. It is important for the user and the librarian to view the same Web 
site or computer screen. Thus, for many libraries, the telephone is limited to ready reference 
purposes. 

There are two assumptions about remote users of library resources. First, remote users 
have different needs fromthose in the library building. The second assumption is that the 
combination of e-mail and telephone reference is not fulfilling these needs because of their 
technological limitations. Archer and Cast (1999) point out that as needs change, reference 
service should evolve to suit new needs and take advantage of relevant technology in providing 
such service. 

In targeting a new service to a sample of users, assumptions were considered such as the 
percentage of existing chat users and the interest in or need for reference help. These 
assumptions, though specific to each target population, should be considered in similar pilot 
projects. 

With limited resources, the project focused on an existing group of library users who 
might already be familiar with online chat software or have basic experience with computers and 
feel ready to explore new software. Moreover, there are different programs to conduct online 
chat, such as Internet Relay Chat and ICQ. For the purpose of this project, ICQ was chosen 
because of its large existing user base and resources available to the author. 

Since ICQ is widely used among high school students, it was assumed that there would 
be some ICQ users within the target freshman population, although there were no data available 
on the popularity of online chat among freshmen at the University of Colorado, Boulder. The 
service was offered to students in conjunction with library instruction in which students are 
expected to fully utilize library research concepts and databases in their assignments. The project 
helped highlight issues and concerns we might have in offering high quality digital reference 
service in real time using free software. 

Setup and Testing 

Before the service was introduced, the author tested ICQ’s potential use and features for 
providing digital reference using a personal ICQ sign-up. Once the author was familiar with ICQ, 
a new ICQ number was set up so that information related to providing digital reference would be 
kept totally separate from personal information. A new ICQ number provided the opportunity to 
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create a user profile that fits the purpose of this exploratory project. For example, users can 
locate the University of Colorado at Boulder digital reference ICQ number by searching the ICQ 
directory. 

Each specific feature of potential use was tested out on the new ICQ account with the 
help of the author’s friends who were also familiar with ICQ. Chat sessions were initiated at 
different times to test out system response rate. Different levels of computer knowledge on the 
part of potential users were simulated through role-playing sessions. 

The new service was introduced to 250 students in spring semester 2000 in selected 
freshmen and sophomore level classes from the Communication Department and Student 
Academic Services Center writing classes. These classes were chosen for the project because of 
existing working relations with teaching faculty. Library instruction was integrated into these 
curricula, and ICQ was offered as one more way for students to contact the author for reference 
help in addition to e-mail, telephone, and regular office hours. 

The author was available during regular office hours (Monday 2:30-3:30 p.m.) and in the 
evenings (until 10 or 1 1 p.m.) in the beginning of the project. After two-thirds of the semester 
was over, when deadlines for research assignments approached, the author’s ICQ account was 
accessible around the clock and students were encouraged to leave an online or offline message 
when real-time assistance was not available or if a chat was in-session. 

ICQ Features for Digital Reference 

ICQ provides a few features that are particularly helpful in providing real-time reference 
service. One such feature is the indication of user availability and the options to customize 
messages being displayed under each availability condition (e.g., available, away, extended 
away, occupied, etc.). For example, when the librarian is occupied, ICQ will display a message 
that encourages users to leave a note or try back again. With offline messaging and e-mail 
features, users can indicate the best time to return their “call,” or librarians can utilize the same 
setup for both real-time and e-mail reference. 

To achieve quality reference service beyond e-mail or telephone reference, software is 
needed that allows multiple channels of synchronous communications. Using ICQ, users and 
librarians can surf the Web together simultaneously. With many online resources now available 
through the Web, librarians can send the Web site address of a specific screen to a user (e.g., a 
specific book record or the screen to start a search). In a similar fashion, users can easily send the 
Web site address to a librarian to show where they were searching and indicate problems seen on 
the screen. This feature is important because by toggling between two separate screens or 
adjusting the size of the Web browser and ICQ chat windows, librarians and users can surf the 
Web or search library resources together simultaneously. McGeach (1999) pointed out the 
advantage of remote application sharing and remote control software. In addition to the ability to 
share URLs, files can also be easily transferred between users. 

Chat sessions also allow users to scroll back to earlier conversations in the same chat 
session so as to clarify or verify information and discussion. A transaction log of each chat 
session can be archived. This can serve at least two purposes: the first one is for research and the 
second is for the creation of a knowledge base. Data can be gathered with the consent of the user. 
Quality of service can also be analyzed for improvement. The author did not explore the 
searching capability of the archive and thus cannot comment on the ease or obstacles in utilizing 
this feature to create a collection of frequently asked questions. 



Issues and Concerns 



Software that is free to both library and target user populations is crucial to the 
accessibility of the new service. On a related issue, software user agreements with many 
restrictions will hinder the quality of service. Other important factors in the choice of software 
include availability on multiple computer platforms (e.g., ICQ is available in PC, Mac, Palm 
Computing), and self-guided installation for new users. 

Privacy 

The issue of privacy commanded a delicate balance on the part of the librarians. ICQ 
allows restriction of any chat request to only other authorized users. This seemed to be a good 
policy until the author discovered that there were problems receiving off-line message (messages 
sent to an ICQ user when the user is not online) with the new ICQ. The authorization process 
relies on requesting and granting authorization through a series of messages. When an ICQ 
account is off-line, it is not possible to grant authorization requests or establish chat sessions. To 
alleviate the problem, the account was logged on 24 hours a day, thus eliminating any off-line 
messages sent to the account. 

By allowing all users to access the chat function, there was an increase in the amount of 
junk e-mail sent to the account from promoters using ICQ. While there was a feature to put any 
unwanted ICQ user on an “Ignore List,” it proved to be a nuisance at times. ICQ does offer 
different levels of security and privacy, and it allows users to adjust those options at any time to 
suit one’s specific needs. 

The ICQ software is equipped with very simple instructions that allow new users to 
initiate a chat a few minutes after installation. It requires far less technical knowledge on the part 
of the user than MOO, yet offers more capabilities than e-mail and phone reference combined 
(Shaw, 1996). Yet, finding support when things do not work is a challenge in ICQ. There is so 
much information on the Web site that it is overwhelming. It does not have a well-organized 
knowledge base. The best strategy is to join other chats groups in ICQ for technical support. 

Staffing 

ICQ does not restrict users to one station. In other words, one can actually be at any 
computer connected to the Internet and still provide reference service using the same ICQ user 
number. The implication is that digital reference allows more flexible staffing possibilities. In a 
consortium, a group of librarians can provide real-time assistance across different time zones. 
When online chat programs are used within a single institution, patterns of use by remote users 
will influence staffing decisions. By working with database designers, network administrators, 
and systems librarians, one can best plan for the effective use of limited resources (Tenopir and 
Read, 2000). 



Marketing/Promotion 

It is helpful to identify computer and Internet use among the target group. For example, 
in this project, data became available towards the end of the semester that indicated few 
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freshmen used online chat or are familiar with the concept. Although it may be possible to do a 
survey of online chat use in several large lecture classes, it may be more efficient to try 
partnering with other campus units that are interested in improving the technology skills of 
students. These might include academic departments moving towards greater integration of 
information technology into the curriculum and information technology services responsible for 
providing primary computer support to students and gathering data for new initiatives. By 
pooling resources and using a larger sample, collected data will be more meaningful and 
accurate. 



Software and Technical Support vs. Reference Assistance 

Real-time digital reference services that require users to install specific software or that 
require complex navigation will be ineffective. Although ICQ is relatively easy to use, remote 
library users who do not already chat online are not likely to install ICQ for this purpose. 
Librarians may also be hesitant to use it. Technical support and customization are unlikely with 
this or any free software. Librarians interested in using ICQ may not want to provide technical 
support for users. 



Other Alternatives and Models 

Alternatives 

Some libraries have tried fee-based chat software that allows easy access to real-time 
digital reference service. One key advantage is the elimination of software installation 
responsibilities on the part of the library user. Instead, librarians rely upon system support staff 
to install and maintain the chat program. 

In addition to chat, instant messaging programs are gaining popularity. While many 
instant messaging programs are not platform-specific, they are not as versatile as chat programs. 
For example, it is difficult to archive instant messages, and complete “chat” logs are not always 
available. 



Models 

Call centers and online customer service methods have impacted overall customer 
satisfaction and loyalty. There are different models of service offered and methods of integration 
in online customer service (Dineley & Snyder, 2000; Steul, 2000). While there are differences 
between the for-profit and non-profit worlds, librarians can learn from the private sector. In 
many cases, companies that just offer a Web presence do not have the resources to invest in the 
infrastructure to provide real-time customer support. Thus, they outsource this component to an 
outside company. This may be a viable solution for libraries. Consortium projects have been 
initiated, and even with current technology, it is only organization culture that forbids us to 
provide a true 24 hours a day/7 days a week real-time digital reference service. Any large-scale 
service project that crosses institution lines requires strong infrastructure in staff training and 
resource-sharing. 

The growing number of trade articles and research papers on digital reference will be 
helpful in the development of future models. Aberg and Shahmehri (2000) conducted a study on 



the use of Web assistants and the importance of adaptability in these programs (including 
different communication style, different expectation or preference for amount and format of 
information, and the delicate balance between computer-based and human-based assistance). 

At the Virtual Reference Desk 1999 Digital Reference Conference, Steve Coffman 
presented a model that screened each request and matched it to different levels of help. 
Collaborative Digital Reference Service (CDRS), a Library of Congress initiative, is creating a 
knowledge base of frequently asked questions as part of a self-help solution; this is a strategy 
used by commercial services as well. Some educational institutions engaged in distance learning 
have started to staff 24/7 help desks, and others have outsourced the provision of such service 
(Young, 2000). Unless librarians are ready to provide what Ann Lipow calls “after-my-library- 
hours-service” (Lipow, 1999, p.52) with little compensation, outsourcing to a group of librarians 
may be the best way to finance a true high quality 24/7 real-time digital reference service. 

Conclusion 

In an ideal world, we can deploy the technology we need to offer valuable service to our 
users. With funding shortages part of our daily lives, librarians can utilize free software and try 
to respond to new needs in a timely fashion. Through the exploration of different software, we 
can formulate a strategy that works for users and librarians. There are other like-minded 
librarians who have tried ICQ for digital reference. The next step is to gather their input and find 
out what works and what does not across all types of libraries. Results of such assessments will 
help us formulate a real-time digital reference service that is both user-friendly and sustainable. 
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Virtual Reference Desk Incubator: A Demo 



Blythe Allison Bennett 

Virtual Reference Desk, Information Institute of Syracuse 

Abstract 

The Virtual Reference Desk Incubator software was designed for start-up AskA 
services that accept, route and answer questions via the Web. This paper describes the 
“front end” for the public view, the “enhanced view” for experts answering questions and 
the “back end” for the administrators of a service. Version 1 of the Incubator is available 
free to non-profit services. Customization costs are borne by individual services. The 
Virtual Reference Desk Project offers training with the software and server space at no 
cost. 



VRD Incubator Software 

Public View 

The Incubator allows patrons asking questions to browse the AskA service’s Web 
site for resources and archived questions in a subject directory or search the site by 
keyword. In order to register to ask questions, patrons must complete a form (see Figure 
5). Later, they can retrieve their answers in a “My Questions” section (see Figure 6-7). If 
a patron has also provided an e-mail address, an e-mail message will be sent when 
questions are answered to alert the patron that an answer is waiting on the site. 

A sidebar section provides highlighted links that appear on all pages. The 
administrator maintains the content and linking for the site. Experts can also access the 
public view. 

Figure 1 illustrates the organization of the categories on the main page. This page 
is created and edited by the administrator using a user-friendly tool in the administrative 
view. 




71 




^ my questions N, seardi archives X help 



“X 



Main 

Sub-Categories: 

Arts 



Biography 



Qsna^fisft aata H s . ahb 

Library Mtihtmftfag 

Ph ygitaLEifoc^isa fe E s ton 

Social Studies Trivia 



O^/O^teACHER 



CoUagt/Cara ai 

Lan ffl iig g A rtg 

Philosophy 

Science 

V ocational Education 



AshA_SEgiljflhi 
AskA Digests 
AaK A+Iocam 

Using AskAs with 
Ti ps for U sin g AskAs 

Experts 

Registration 
Question PickuR 



Logout 




Fig. 1 . Public view: main page. 



Within each category on the main page are sub-categories (Figure 2) created by 
the administrator according to the needs of the audience. 




home \ search archives N ftetp 



l>^l>^jeaOTe 



Main : Science 

Sub-Categories: 

Anatomy Science Astronomy Biography Science 

Biology (Hum anl Biology and life Sciences Botany 

Chemistry Computer Science EerthSciencs 

Ecology Science Engineering 

General Science 

H eUual Historvi-Pale^ltology Qgeapo g iaphy 
Physical Science Physics 

SflSQaeJUk Space Sciences 

Zoology 




Meteorology 

Paleontology 

Sci-Fi 

Technology 



AskA_ScQlliaht 
AskA Diaesls 
AskA+ Locator 
*J sin a As kAs w i t h 

students 

Tips for Using AskAs 



Experts 

Req i> t n>tign 

Question Pickup 




Fig. 2. Public view: categories. 



Within each sub-category are two tabs: resources and archived questions. Editing 
in each section is easily done in the administrative view. The resources section may be 
used to list Web sites or other types of resources. The resources appearing in Figure 3 are 
AskA services related to the sub-category “oceanography.” 
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Fig. 3. Public view: resources. 



The archived questions collection (Figure 4) includes questions from the database 
of previously answered questions. Each entry in the archived questions tab points to the 
corresponding answer archived in the database. 
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Fig. 4. Public view: archived question. 



Registered Patron View 

As previously mentioned, patrons who wish to submit questions to the service 
must register (Figure 5). Only first name (or nickname) and grade level are posted on the 
public view when the answer is added to the database. Patrons may also include an e-mail 
address if they wish to receive notification when an answer is ready at the My Questions 
section of the site. 
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Fig. 5. User registration. 

When the patron wants to retrieve an answer to a question on the “My Questions’ 
page, he or she must log in (Figure 6). If the patron forgets his or her password, there is 
an option to have it e-mailed if an address was provided. 
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Only patrons who have registered through the VRD learning Center signup 
will be able 1o ask a question. Registration is free. Click here to register . 

Click here if you do not remember your password, or you are having difficulty 
logging in. 
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Fig. 6. Patron view: “My Questions” login. 

Once logged in, the patron can view all the past questions asked under that 
account. Unanswered questions are included with a note stating no answer is available yet 
(Figure 7). 




Date: 9/1/00 

Category; General Reference 
Question: 

I hear green tea is supposed to be good for you, why is that? 

Answer 

--Green tea is thought to have many benefits, such as: fighting viruses, 
slow aging, contains vitamins C, E, and beta carotene, Weight-loss 
experts believe that green tea may promote the burning of fat. Yay! S.Q. 
Susan Quigg - 9/4/00 



Date: 8/2/00 
Category: Science 
Question: 

how much coffee does it take to wake my mom up in the morning? How 
■ much is in espresso coffee? 

'rtu^wei , 

—Hi Allison! The ingredient in coffee that helps your mom wake up is 
called caffeine. You can leant more about it at some sites that I found by 
using the search engine Google at: http jfwww. go ogle, com and searching 
for caffeine Coffee and Caffeine's Frequently Asked Questions 
http^Avww.landfield.comffaqs/caffeine-faq/This page has all kinds of 
information, including the amount of caffeine in various products. 
Frequently Asked Questions about Caffeine 
httpy/coffeefaq.com/caffaq.htmlThis rite also has lots of information 
including how much caffeine is in different kinds of coffee. I hope these 
sites will help! Also check in your library for more resources. Thanks for 
using the Virtual Reference Desk! Dorothy 
Dorothy Phipps - 9/4/00 



Date; 7/28/00 
Category: Earth Science 
Question: 

I need to find some pictures of the great rift valley to show the different 
plates colliding. Where can I find some? 

Answer: 

There are no answers yet. 



Fig. 7. My Questions archive. 



Expert View 

Experts (individuals who answer patron questions) are permitted to log in via the 
public view but are not allowed the full range of administrative permissions. First-time 
experts register through the sign up form in the public view (Figure 8). On this form, 
experts can select the categories in which they wish to accept questions (e.g., arts, 
science, etc.). The administrator must approve experts before they may log in to the 
Question Pickup section. 
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Fig. 8. Expert registration. 



Registered experts log in by entering the Question Pickup area (Figure 9). If there 
are any questions in the queue to be answered in an expert’s subject area, the questions 
will appear. The following example shows questions in social studies, general science, 
and astronomy. The expert can then select a question to answer. 
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Biology and Life Sciences: 

There are no current unanswered questions. 
Chemistry: 

There are no current unanswered questions. 
Ecology Science: 

There are no current unanswered questions. 
Engineering: 



Fig. 9. Expert view: questions to answer. 



The expert uses the screen shown in Figure 10 to compose an answer to the 
selected question. Once the answer is typed in and submitted, the patron receives an e- 
mail message saying an answer is waiting. The question and answer pair become part of 
the database to be searched and appears on the archived questions tab for that category. 
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Fig. 10. Expert view: answering a question. 



Administration View 

There is a dynamic administrative tool that controls all the linking, content, 
categories, routing, editing and other functions (Figure 1 1). This section is only available 
to administrators via a password-protected login and a hidden login page. 
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Fig. 1 1 . Administrative tool. 
The tools available are: 



New Questions - Administrators can view the new questions and sort by subject or by 
date. Options include edit, keep, send, and delete. “Edit” allows administrators to change 
a category and subject heading as well as edit the question. Administrators can then 
“keep” the question internally for a local group or “send” it to an external group. 
Administrators can navigate through the questions by using the next and previous 
questions buttons. 

Question Queue - These are questions that are kept internally. They will show up on the 
screen for an internal expert who has selected that category. 

Categories - Administrators can edit, add, or delete categories, topics and subtopics. 

AskA Service Profile - Administrators can add, edit, and delete profiles of AskA services 
(this function can be adapted depending upon the needs of the service). 

Q&A Archive - In addition to actual questions and answers going through the system on 
an ongoing basis, administrators can also add entries here by filling in a question and an 
answer and then submitting the information. Editing is also available. 

Administrators - This function allows administrators to add, edit, or delete 
administrators to the system. 

Experts A waiting Approval - Here, administrators can edit categories, view information, 
and delete or approve experts. When “approve” is selected, the administrator is taken to 
the registered experts page. 



Registered Experts - Choices are edit categories, view information, and delete experts. 
Users - Administrators can edit or delete user accounts. 

Outside Experts - Options are to edit information (by going to “view info”), deleting an 
expert’s name, or adding experts (at the bottom of the page). 

External Expert Question Queue - This area is for experts who are outside of the system, 
Options are: Question Options, Mail Question, and Keep Question (for an internal 
expert). Under Question Options, choices include: Return question to “New Questions,” 
keep this question internally, move the question to a different queue (i.e., change it to a 
different external expert), mail the question, edit the question text, or delete the question. 

Expert-Answer Details - Administrators can view the expert ID, expert’s name, number 
of questions answered, and under question details, view the questions and answers for all 
that expert’s previously answered questions. 

Question-Expert Details - Administrators can view the question ID, date, question, 
expert who answered it, and the answer. 

Registration Details - Administrators can add, edit or delete categories. They can edit 
user registration information and the white side bar information by changing or adding 
menu names and links. 



Contact Information 

Virtual Reference Desk 

http://vrd.org 

vrd@vrd.org 

621 Skytop Rd. Suite 160 
Syracuse, NY 1 3 244 



1-800-464-9107 (phone) 
1-315-443-5448 (fax) 



The Question Board 



David Ward 

University of Illinois Undergraduate Library 

Presentation 

Introduction 

How does fabric softener make your clothes soft? Was Sam the Butcher (Alice's 
stud muffin from "The Brady Bunch") Magilla Gorilla's voice? While many students toil 
long nights at the library researching traditional term paper assignments, students at the 
University of Illinois’ Undergraduate Library have long pursued more esoteric 
knowledge, courtesy of an anonymous reference service known as the Question Board 
(QB). This service allows students to submit their questions via paper or e-mail and then 
view their answers on a paper or electronic “board.” QB provides a forum for the library 
staff to answer some of life’s tougher and more mysterious questions, while at the same 
time introducing undergraduate students to the library’s reference services and 
collections. 

This presentation traces the history and development of this reference tool, from 
its inception in the early seventies, through its entrance into the electronic age in 1997 to 
its current state. It explains the technological underpinnings that have given this virtual 
reference tool life on the Web, from both staff and user points of view, and includes some 
sample ASP and Visual Basic scripts that are used to access the online archive of 
previously answered questions. The presentation also includes a description of the Web 
interface that students can use to ask questions; sample student questions are included 
throughout the presentation. Finally, it explains how QB functions as a public relations 
tool for the library and an online gateway to the library’s other reference services. 
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^Reception and Role in the Library 


1 




2 



? ? 




? Baclkgifomai ? 






#What is QB? 


Where did Grape Huts 

? Get its Name? 




- Anonymous (non-confrontational) form 

- htto://www. library. uiuc.edu/uql/qb/ 






$How does it work? 


1 




- Questions submitted online or in paper 

- Answered by library school graduate students 

- Answers posted to paper board or Web site 


3 




4 
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? History ? 




? ? 


* Started circa 1973 (paper bulletin-board) 




Does chewing gmm 
really stay in your 


* DB2 and Word Perfect files started -1989 




stomach for 7 years 


0 Converted to Microsoft Access/SQL 1997 
$ Made Web-accessible 1997 




when you swallow It? 

6 ? 


5 




6 



? Technological ? 




7 Technological ? 


Backbone 




Backbone 


#Staff Interface - Microsoft Access 




®Links to SQL Database 


- Data Entry Form (Fiqure 1) 




- Linked Tables (Flaure 3: SQL Keyword Table) (Fiqure 4: 


- Reports (Fiqure 2) 




SQL QB Table) 






— Updatinq (Figure 5: Access Queries) 






— Keyword Macro (Fiaure 6: Access Module Code) 


7 
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? Teehs&otogio&l ? 

Backbone 




? ? 


^Patron Interface - The Web 

— Demo : http://wwwJlbrarv.uiuc.edu/uQl/qb/ 

- Search vs. Browse modes 




Blow does febiic 
softener make y©w? 
clothes soft? 


- HTML. ASP. and VBscriot (Sample Script - Flaure 7: 






QB ASP Code) 






9 
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7 Reception and Role ? 

within the Library 




? Reception and Role ? 
within the Library 

^Common Types of Questions 

- Campus and University of Illinois History 




^Student Uses 




- Celebrities 


- Diversion / Procrastination 




- Movies 


- Non-Threatening Source of Information 




- Sex / Relationships 


- Entertainment 




- Drugs 

- Personal Problems 


1 




- Library-Related Issues J 


1 




2 
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? Reception end Role ? 




? Reception and Role ? 

within the Library 


within the Library 




« Advertisement for Reference 




#Problems/lssues 


Services 




- Workload 

- Turnaround Time 


- Promotes the Undergraduate Library Mission 




- No Reference Interview 


- Accepts All Questions (non-judgmental) 

- Highlights Reference Collection 




- Not all Questions Have an Answer 


- Generates Articles in Local Press “J^ 




1 


3 
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? ? 
Implications and Fntnre 
Directions 
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$ Portability of Database Technology 




♦ Pittman, D. (1987) The Question Board. College and 
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- ExamDle: www.askieeves.com 1 
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PARLREF: Digitizing the Reference Request Process 

Kate Whitridge and Margo Jeske 

Information and Documentation Branch, Library of Parliament, Canada 

Abstract 

This paper features PARLREF, the intranet-based automated inquiry management 
system used by the Library of Parliament. It outlines efforts to automate the library’s 
workflow process and develop substantive question and answer repositories. The authors 
discuss how the library is moving towards a bilingual corporate knowledge management 
environment for research and reference professionals through PARLEF. 

Introduction 

The Library of Parliament, established at Confederation in 1867, offers the range 
of professional and personalized services that Parliament needs for responding to the 
challenges of the information society. The library provides information, documentation, 
and research and analysis services to 105 Senators and 301 Members of the House of 
Commons. Information about Parliament and visitor and education services are also 
provided to the general public. 

Each year, the library’s Public Service Division handles over 140,000 information 
and documentation requests with very tight deadlines for parliamentary clients and for the 
general public. Requests are received at seven different locations; in this distributed 
. service environment, an electronic solution for sharing information on client requests is 
of vital importance. Intranet technology offers the additional opportunity to develop 
request forms that the clients may submit directly from their desktops. 

The library began looking for an automated system for tracking and managing 
these requests with Axcom Texto (Famic) in 1995. In 1996, it was decided that the 
PARLREF system must be a secure, bilingual, intranet-based workflow and information 
management tool, complying with software and platform standards in place for the library 
and the House of Commons. WorkLogic.com Corporation was chosen as the developer in 
1997. 



PARLREF 

PARLREF operates as an online request system that supports the recording of 
client requests; assignment of questions to appropriate staff; division of requests among 
different sections of the library; and monitoring of question progress through receipt, 
assigning, and response. The inquiry-recording, tracking and related components are 
available to staff through the library’s institutional intranet site, while request submission 
functionality is available to authorized clients through the parliamentary intranet. The 
client submission forms permit parliamentary clients to transmit their requests securely 
from their desktops, during and outside regular hours, and a link through the library’s 
Web-based catalogue enables them to request books or articles. 

To preserve strict confidentiality, PARLREF has three levels of security. Use of 
the PARLREF Web site is restricted to those who have a valid network user name and 
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access to the parliamentary intranet. Furthermore, to access PARLREF, the user’s name 
must be entered in PARLREF’ s user database. PARLREF authenticates network users 
against its database, and if unsuccessful, users have the opportunity to log on with a 
specific PARLREF user name and password. Then, within the PARLREF application, 
there are three user levels: clients have a basic level, allowing them to only create and 
submit their own requests; library staff have a working level, allowing them to create and 
manage requests; and managers and the PARLREF team have a third level of access, 
allowing them to also manage staff workload and produce statistical reports. 

The PARLREF system consists of three modules (Figure 1): PARLREF itself, 
where requests are created, distributed and managed; OrgSite, the database for user and 
client information; and, DocuSite, the electronic document management system. While 
PARLREF has its origins in a help desk request management application, it has been 
substantively customized for a distributive reference desk, and some development work 
has been done to customize the DocuSite software. A fourth module for schedule 
management is under consideration, and will assist in the assignment of requests by 
supplying information on staff availability. 




Fig. 1. Structure of PARLREF. 



Forms 

Incoming requests are recorded in PARLREF’s front end. Requests are entered on 
three forms or tabs: Question, Client, and Sources/Documents. A fourth tab called 
Transfers & Notes, available in the Management Tool, lists all transactions related to 
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existing requests as they are processed by library staff. The question form (Figure 2) has 
profiles that are customized to the particular type of service offered by the library 
(reference, research, general public information, and document requests). Both question 
and client forms have certain mandatory fields, and a mixture of fields that are free-text 
or have predefined lists of values (with defaults where acceptable). 
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Fig. 2. PARLREF question form. 





The client form (Figure 3) is quite flexible; the client database may be searched 
for parliamentary clients, and the retrieved data may be revised or new data entered on a 
blank client form. A field for modifications on this form identifies whether the request’s 
client form contains data that needs to be verified. Flagged requests can be retrieved, the 
information checked, and the necessary changes made to client records in the OrgSite 
database. 
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Question 



Client , 



<— _ 

Last name: [ 

Organization: | 

Salutation: j 

Gender: O Male O Female 

Telephonel: [ 

Fax: 1 

Room : ■ | 

Address: | 

Province/state | " 

Postal | — 

code: * 



\ Sources/ Documents- >f Transfers & Notes ) 

f 



First 

name: 



Suffix:.. 
Language: • 
Telephone2: 
E-mail: 
Buildings 
: i: City: 
Country: 
Multi lis: 



[ 



O English O French • 



Changes: <?; Temporary O Permanent (to be Verified) 



Search 



Print 



Label 



Remarks: 



r 



Question ; 



1 



Client 



. .^Sources / Documents : ;[ Transfers & Notes j 



Fig. 3. PARLREF client form. 
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Document Management System 

Through its integrated document management system, DocuSite, PARLREF 
allows staff to: 

• associate electronic documents with particular information requests, 

• distribute online information files, 

• share electronic documents frequently consulted by staff or provided to clients. 

DocuSite operates as an online document publishing and management portal, with all 
access via the Web browser. Electronic documents are linked to specific requests on a 
request’s sources/documents form. Supporting documents received by fax or mail can be 
scanned and attached to requests - a particularly useful feature when these requests are 
then divided among staff across different divisions or at diverse locations. 

Documents uploaded to DocuSite are stored in collections and can be retrieved by 
collection or subjects within those collections, or by searching. In addition to the 
collection of request-supporting documentation, DocuSite collections include FAQs 
prepared by staff, common or current reports (full-text where permissible or brief 
abstracts for print-only versions of “hot” documents), and pamphlets, kits and lists related 
to Parliament. 



Procedures for Client Use 

Parliamentary clients of the library may submit their requests using the Intranet 
Online Request Service (IORS) forms or request a document using a link from the 
library’s Web catalogue. Clients select the library service that they require and enter their 
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inquiry on the corresponding request form: Information/Reference, Research/ Analysis, or 
Borrow/Copy Document. The catalogue interface transfers the document data from the 
catalogue record into the Borrow/Copy Document request form. These IORS request 
forms are based on the principal service forms accessible in the staff version of 
PARLREF, with the addition of simple instructions. 

Before clients submit their requests, their contact information from the OrgSite 
database is displayed, and they may identify any changes to be made to this information. 
On submission, they are given a reference number which they may use in subsequent 
communication with the library. The clients may return to the intranet or to their 
catalogue search, or may submit another request while they are still logged onto the 
system. As illustrated in Figure 4, there are provisions for clients to log on or request 
access to PARLREF in the event that their network user information is not immediately 
recognized by PARLREF. 




Management of Requests and Responses 

In PARLREF’s Management Tool (Figure 5), staff can query the database of 
existing requests and track requests from creation to completion. Staff can access their 
requests by the status of request, or can search by client, keyword, dates, or combinations 
of various fields. Managers have additional rights to search the database for the requests 
assigned to specific employees or sections and have access to a report generator that 
produces statistics on requests, clients, and documents. 




89 



ParIRef Version 0.81 



Mm 

npnni 

sill 



Management Tool 



m 



My Requests: 

•new - 1 
• active - 3 

Choose a query: 



My New Requests 
My Active Requests 
My Closed Requests 



Frangais 



Logoff 



Katherine Whitridge, Reference 



Query -> 



Fig. 5. PARLREF Management Tool. 



New Requests 
Request by Reference Number 



Clients to be Verified 
Management Query 
Requests by Employee 
Requests by Org. Unit 



Requests can be transferred or copied to various library organizational units. 
PARLREF enables sharing of requests among reference librarians, technicians, and 
research officers, and allows for coordination of internal requests to the collections unit 
for acquisition of a government document or new book. A search on divided requests will 
display a hierarchical list of the requests. 

The PARLREF request database is a valuable knowledge base that supports 
reference work as a dynamic and interactive process where questions and answers require 
collaborative expertise among individuals interacting with information. The system 
permits a widening of the view of information to a thoughtflow process, with the ability 
to track key intelligence. For complex or difficult reference or research requests, staff are 
encouraged to specify the resources that they used to respond to the question, recording 
print resources in a ffee-text field on the question form, and attaching electronic 
documents or representations of the documents to the sources/documents form. Staff can 
search for requests by keywords from the field where the question is described in detail, 
identify similar requests, and consult the sources used by other staff members or retrieve 
documents from DocuSITE. 

PARLREF’ s request receipt and assigning workflow reduces active duplication of 
effort. For instance, the staff who respond to the telephone line create requests, and then a 
member of the PARLREF team assigns those requests to appropriate staff. The 
PARLREF team member is in a position to view all requests received at all service 
points, can pinpoint trends in key new issues as they emerge, and can coordinate the 
assignment of questions for the most thorough response and efficient use of the library’s 
resources. 



Planning and Implementation 

In 1999, the two librarians working part-time on the PARLREF project team were joined 
by two more librarians. Team members’ responsibilities for the various system 
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components were defined, and an accelerated schedule of workflow analysis, 
specification design for the customized software, and preparation for training and 
implementation of the PARLREF system commenced. The library’s PARLREF team 
works closely with WorkLogic.com Corporation in development activity, and is assisted 
by two programmer/analysts from the House of Commons who also administer the 
system. Initial training and testing by a focus group of librarians and technicians in the 
Public Service Division took place in early 1999, followed by further intense system 
development. 

Phase I of the implementation of PARLREF began in October 1999 with the 
replacement of a tracking system in Information Services that was not Y2K-compliant. 
The PARLREF team librarians were assigned full-time to the project. Attention was 
given to the infrastructure required for PARLREF, including typing tutorial software, a 
telephone monitoring system, and headsets. In November 1999, Information Services 
staff were trained on the new system and entered requests in both tracking systems for a 
week; the results were evaluated by the PARLREF team. On December 1 , PARLREF 
was implemented in Information Services. Within six weeks, staff use of the new system 
had become instinctive. The library’s video service now also uses PARLREF. Further 
training for initial user groups, in the form of meeting briefings or in-house mini-sessions, 
has been designed to accompany major upgrades to PARLREF. After six months of use, 
over 19,000 requests had been created; the volume will magnify substantially as other 
reference services come online. 

Ongoing support for the system is provided by PARLREF team librarians, and an 
e-mail service account facilitates the communication of questions and problems from 
PARLREF staff users to the team. Support duties are shared among team members, with 
problems related to specific system components referred to the appropriate member. The 
e-mail account also supports the sharing of documentation and provides a development 
forum for the team. 

In the spring, as Phase II neared (i.e., implementation of PARLREF for full 
reference services), the PARLREF team and managers of the Public Service Division 
began a series of planning and policy meetings; these meetings were designed to ensure 
that documentation on PARLREF could be augmented with policies and operating 
procedures. Several librarians and library technicians were identified to participate as 
PARLREF team members on a limited basis. 

The next phase in the library’s implementation process will involve the 
Parliamentary Research Branch. A group of researchers has been selected and will 
become active in the customization and preparations for extension of the system to the 
complex services provided by that branch. 

Ongoing projects include identifying, and customizing as necessary, scheduling 
software to support the request assignment process, and developing the archiving 
functionality for the PARLREF databases. In September, the team will begin planning for 
Version 2 of PARLREF’ s central component. Among the items considered for future 
development are: 

• more powerful methods of handling requests that have multiple request types, 

• redesign of some of the screens for ease of use, 

• more sophisticated Boolean searching by keywords, 
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• additional sorting keys for request lists. 

Conclusion 

The library has developed a very powerful and flexible automated request tracking 
system with PARLREF and has begun the work of creating a valuable knowledge base of 
requests. In the coming year, the library will be temporarily vacating its beautiful, 
historical home for a period of renovation. PARLREF will play an integral role in 
ensuring that the transition is seamless to the library’s clients and will greatly facilitate 
request-related information-sharing among staff at many service points and across the 
library’s organizational structure. 

Note 

All screen shots of the PARLREF system are used with permission from the Library of Parliament, Canada. 




92 



Digital Reference for Florida’s Distance Learners 

Meredith Ault and Rachel Viggiano 
Florida Distance Learning Reference & Referral Center 

Presentation 

Introduction 

This presentation addresses issues of staffing and workflow at a statewide digital 
reference service and the challenges associated with providing electronic reference 
support to distance learners. Other topics include the evolution and evaluation of the 
Florida Distance Learning Reference & Referral Center’s Web site, Web forms, and 
services. 
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Florida Distance Learning 


Digital Reference for Florida’s 




Reference & Referral Center 


Distance Learners 










1 • Statewide agency funded by the legislature 






j • Provides library and research services to 


Meredith Ault & Rachel Viggiano 




distance learning students and faculty 
i • Part of Distance Learning Library Initiative 


Florida Distance Learning Reference & Referral Center 




(DLLI), a cooperative effort of the state 






j universities, community colleges, public 


Virtual Reference Desk 2000 j 




s libraries, and the State Library of Florida--™^ 

! ^ 


October 16-17. 2000 
Seattle. WA 




i 




Florida Distance Learning Reference & Referral Cantar 



Florida Distance Learning 
Reference & Referral Center 




i 

/* ■ 

History of RRC 


• Serving students and faculty at 73 Florida- 
based regionally accredited colleges and 
universities. 




1 • RFP was issued by DLLI 
i - University of South Florida won grant 

- RRC housed in the Tampa Campus Library 


- 10 state universities 




• Opened in December 1997 


- 28 community colleges 

- 35 independent colleges and universities 






Florida Distance Laarning Reference & Rafarral Cantar 




! 

Florida Distanca Laarning Rafaranca & Rafarral Cantar 



<5 p://www.rrc.usf.edu/pres/vrd/index.htm 

ERJC 
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RRC Hours 




• Seven days a week 




• Over 100 hours per week 


Monday-Thursday 


7:30 am - 1:00 am 


Friday 


7:30 am - 9:00 pm 


Saturday 


10:00 am - 8:00 pm 


Sunday 


Noon - 1 :00 am ^ 






Florida Distance Learning Reference A Referral Center 




RRC Staff 




• 5 full-time librarians 

• 2 part-time librarians 

• 2 graduate assistants 




Photo ua*d with p«rmts*k>n from th« Florid* Distance Learning Rsfarsnca A Referral Center . 
Florida Pittance Learning Reference A Referral Center 



RRC Services For Students 

• Ready reference 



• Research assistance 

• Basic technical support 

• Assistance with locating materials at local 
academic and public libraries 

• Referral to information about distance learning 
programs and courses 



Florida Distance Learning Reference A Referral Conter 



Student Use of RRC Services 




■ Ready Reference 

■ Research Assistance 
0 Document Delivery 

■ Technical Support 

□ About RRC 

□ About Libraries 

□ About Courses 



7% 

Florida Distance Learning Reference A 



Referral Center 



O’ p://www.rrc. usf.edu/pres/vrd/index.htm 

ERIC 
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RRC Services For Faculty 




Staffing Issues 


• Onsite, online or broadcast library instruction 

• Creation of course-specific Web pages 

• Brochures and other print materials outlining 
services 

ss 




I • Staggered schedules 

' - 7:30 am - 4:30 pm 

- 9:00 am - 6:00 pm 

- Noon - 9:00 pm 

1 - 1:00 pm • 10:00 pm 

j • Graduate assistants for late nights 
j - 9:00 pm - 1:00 am 

; • Flexible during vacations, conferences, offsite 
instruction 


Photo u»*d with permission from the Florid* Distance Learning Rafarence 4 Rtferral Center. 

Florida Olstance Learning Reference 4 Referral Conler 




Florida Otslanco Learning Roforonco & Referral Center 
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Method of Contact 




Method of Contact 


• Ways in which students contact the RRC: 




xtP\ 


- E-mail 

- Web form 

- Chat 

’ - Toll-free phone 

- Fax, postal mail, walk-in 




(3 ) /dP\ 

Vi *98 y DPhone \ 

□ E-mail 

DWebform l 13# /° J 

■ Other Xs' 66% j 






\/l999 / 2000 


Florida Olstanee Learning Reference & Referral Center 




Florida Oiitance Learning Reference 4 Referral Center 
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Workflow Issues - E-mail Contact 

E-mail - rrc@lib.usf.edu 

• All staff members receive e-mail via a 
distribution list. 

• Whoever is "on" is responsible for answering 
1 question. 

|# Answers copied to distribution list for tracking 
purposes. 
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Workflow Issues - 
Web Form Contact 



! Ask-A-Librarian - www.rrc.usf.edu/ask.html 
I • Web form output sent as email to 
| rrc@lib.usf.edu. 

I • These questions are handled the same as e- 
| mail inquiries. 

! • This method of contact is preferred over e-mail 
because it elicits more “reference interview” 
information. 



Florida Oislanca Looming Rafarenca & Referral Centar 
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Result of Web Form 
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Workflow Issues - Chat 

RR Chat - www.rrc.usf.edu/chat 
: • Only two librarians log in at once 
■ - don't want to overwhelm patrons 

, - cover breaks 

\ • Staff alternates logging into chat room. 

• Bell alerts RRC staff when a visitor enters chat 
room. 

j Florida Distance Le arning Reference & Referral Center 
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I Workflow Issues - Phone 



| Toll-free phone 

»• Available staff answers call. 

. • Voice mail used for after-hours calls. 

I • Calls queue when line is in use. 
j • Brief answers given over the phone, more 
detailed responses are sent via e-mail. 



Florida Dtolanca Laarnlng Reference & Referral Center 



Challenges 




| • Providing technical support for library resources 

- Numerous universities and policies 

• varying degrees of academic computing support 

• different IT systems 

- User unable to differentiate between computer, ISP, 
browser, etc. 

I - RRC staffs different levels of computer 



Florida Distance Laarnlng Reference & Rafarral Cantar 
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Challenges, Cont. 




Challenges, Conf. 


i • Providing the best reference assistance 
; - Difficult to conduct a reference interview 

- Differing skill levels of patrons 

! - Extra emphasis on full-text sources 

- Different resources at each institution 

i 

i 




• Providing accurate information about services 
- Reciprocal agreements 
j - Interlibrary loan policies 
;• Improving awareness of distance learning 
i issues 

!• Territorial disputes 


! 

! Florida Distance Laarnlng Rafaranco & Rafanal Cantar 




i 

i 

Florida Distance Laarnlng Rafaranca & Rafarral Canter 
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Challenges, Cont. 




Successes 


• Providing information about distance learning 




1 • High user satisfaction survey results 


courses 




i - user satisfaction surveys conducted yearly 


- no centralized statewide catalog 




• Adding more staff 


• Funding unstable since inception 






i • Increase in number of patrons 
• Implementation of chat software 

I 


Florida Distance Learning Reference 4 Referral Center 




t 

Florida Distance Learning Reference 4 Referral Center 



What We’ve Learned at the RRC 




! 

j 

What We’re Looking Forward to 


• Constant revision of Web site and Web forms is 




: • Expanding chat service 


necessary. 




! - increased hours 


• Marketing of RRC services is key to success. 




! - library instruction for classes 


, • Difficult being only people associated with this 




• Streaming instructional video 


project on a full-time basis. 




: • Vitalization of Florida Virtual Campus project 


Florida Distance Learning Reference 4 Referral Center 




Florida Distance Learning Reference 4 Referral Center 
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Interactive Reference Project - Assessment After Two Years 

Sam Stormont 
Temple University Libraries 

Abstract 

This paper presents a brief history of the Temple University Libraries’ real-time 
reference service and describes the challenges encountered in launching and maintaining 
the Interactive Reference Project. The major issues that emerged during the initiative are 
discussed and include staffing, promotion, technical support and the effectiveness of this 
mode of service. Technical challenges are also addressed, and a distributed staffing 
model is described for the TalkNow service. Factors that contribute to success are 
outlined and include management and staff support and an atmosphere that encourages 
people to become involved and invested in the service. 

What is Real-Time Reference? 

The goal of the Temple University Libraries’ real-time reference service is to 
allow patrons to submit reference questions electronically and receive immediate 
responses. This service provides a response within seconds, as contrasted with e-mail 
reference, which can take hours or even days. 

History of TalkBack and TalkNow 

In November 1998, Temple University Libraries launched a new real-time 
reference service called TalkBack as a pilot project. This service allows library users to 
connect directly with staff at the Paley Library Reference Desk through a link on the 
libraries’ Web site. The user types a question into a Web-based form that includes name, 
e-mail address and comment fields and then clicks on a “submit” button to transmit the 
question to the library. A librarian receives the question and types in a response. The 
exchange is much like a chat program interaction. 

We received positive feedback about TalkBack, and students began using the 
service without publicity or promotion. We simply put up some links on the library Web 
pages, crossed our fingers, and waited to see if anyone responded. The lack of publicity 
allowed the reference staff time to get used to TalkBack. Articles were published in the 
library newsletter and the computer services newsletter, but we remained otherwise low- 
key. As the pilot project progressed, we identified some shortcomings and set out to find 
ways to overcome those drawbacks and improve the service. 

We researched existing software by reviewing academic and trade journals, 
monitoring newsgroups and electronic discussion lists, and consulting librarians and 
faculty. No existing software package met all our criteria. Two students in Temple’s 
Computer and Information Sciences department developed a prototype based on our 
specifications. During the summer and fall of 1999, we tested the service and then 
launched Temple TalkNow on December 20, 1999. 
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How TalkNow Works 



Temple TalkNow uses the Linux operating system (a version of Unix) and a 
scripting language called PHP. PHP is a server-side, cross-platform HTML-embedded 
scripting language that allows Web developers to write dynamically generated pages 
quickly. 

Here’s how a TalkNow transaction works: 

1. The librarian comes to the desk and logs on to TalkNow. She’s ready to receive 
questions at the workstation. 

2. A user clicks on the TalkNow link, and is connected to the dedicated TalkNow 
server. 

3. The TalkNow screen appears, telling the user that a librarian is logged on and 
explains that the user can type in his question. 

4. When the user is ready, he submits the question by clicking the “send” button and 
the question appears on both his monitor and the librarian’s monitor. 

5. The ensuing conversation proceeds like a standard chat dialogue. 

Proliferation of Chat and Instant Messaging Software 

Chat and instant messaging software have become immensely popular. Over one 
billion instant messages (IM) are sent over the Internet each day, and three million users 
are signing up for free public instant messaging services each month (Miller, 2000), 
making IM one of the fastest-growing Web-based technologies. ICQ (“I Seek You”) and 
AOL’s Instant Messenger products have over 153 million users (Kane, 2000). In the past 
year, a number software packages have become available that allow companies to 
provide live customer service over the Web. 

Libraries are beginning to experiment with live reference service. The University 
of Leicester Library has created an excellent site for their Elite Project 
(www.le.ac.uk/li/distance/eliteproject/project/elite.html), with links to many electronic 
and real-time services and a wealth of information on the subject. 

Issues in Providing Real-Time Service 

At Temple University, several major issues have crystallized over the past two years 
including: staffing, promotion of service, technical support, and service effectiveness. 

Staffing 

Real-time reference is labor-intensive. It represents an additional access point 
competing for librarians’ attention. Since our reference department was already stretched 
thin, all staff did not welcome an additional service. Another issue is the perennial 
problem of uneven, unpredictable patron demand. If the reference desk is not busy, it is 
not problematic to answer an online reference question. However, if there are walk-in or 
phone patrons, the librarian must juggle requests. We considered moving the TalkNow 
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service away from the reference desk, but decided against it, since this would require 
assigning a professional to it, putting even greater demands on staff time. 

Librarians were initially skeptical. Several considered a real-time service just one 
more thing that would create stress at the desk, another stream of questions that would 
make it more difficult to provide quality service to walk-in and phone patrons. Some 
simply objected to the chat mode, finding it limiting (e.g., “If I could speak to him on the 
phone, it would be easier and quicker to explain.”). Many people were unfamiliar with 
chat technology, which may have caused them to be wary of this type of service. 

However, everyone agreed that learning to use the software was relatively easy. The hard 
part was making it work for a satisfactory reference transaction. This works best when 
both the librarian and student are motivated and patient and there is plenty of time for 
both parties to focus on the transaction. Transactions are less successful when several 
minutes are required to find information for a user or it is necessary to ask a number of 
clarifying questions; in these cases, the user sometimes gets impatient or technical 
problems interfere with the completion of the transaction. 

One possible solution to managing the extra demands on staff created by real-time 
reference is based on the concept of the distributed staffing model, where librarians at 
multiple locations share the question load. The unpredictability of demand within a 
defined time period is one of the most vexing challenges facing any reference 
department. For instance, one given hour in which two people staff the desk may include 
moments when one or both librarians are not actively engaged with a patron as well as 
moments when four people show up at the desk simultaneously. 

Temple has several departmental libraries located in various buildings on the main 
campus as well as libraries on other campuses (Ambler, Tyler, and Harrisburg). Staff 
from several different libraries can log onto TalkNow simultaneously. This increases the 
odds that when one desk is busy, another has staff available to answer a particular 
question. 

Another possibility for managing staff workload is referrals. For instance, a question 
taken by someone at the main reference desk may decide it is more appropriate for the 
engineering library and will refer the question to the engineering desk. 

Another very important factor related to staffing is an individual’s attitude toward 
technology. Staff who embrace technology and consider it fun and interesting will fare 
much better in a real-time reference environment than those who learn and use new 
technology reluctantly. The more people willing to adopt use of the new technology, the 
greater the likelihood for success with real-time reference. 

Promotion of Service 

A second major challenge is advertising a service. E-mail reference in academic 
institutions has consistently drawn a small handful of users as compared with walk-in 
users. Real-time, online reference at Temple University is following the same pattern of 
only a few questions a day, although we know that many people are searching and using 
our Web pages. It’s unlikely that users are finding everything they need easily and 
quickly on the Web pages, and therefore do not need reference help. It is more likely that 
the service has not been sufficiently publicized. Temple’s publicity efforts have included 
publishing articles in the library and computer services newsletters; promoting the service 
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to supervisors of all the campus computer labs as well as all the staff in our main and 
branch libraries; and including multiple links to TalkNow on the libraries’ Web pages. 
However, the use of the service is still quite modest. 

With the redesign of the library home page this fall, the TalkNow link will be 
positioned more prominently (it is now on the home page, but users must scroll down to 
see it). The link will also be included on additional pages throughout the library Web site 
to increase TalkNow’s visibility. We have also considered conducting a contest to 
increase awareness. In addition, the library has initiated a partnership with the Online 
Learning Program at Temple, which we hope will increase campus awareness of the real- 
time reference service. 



Technical Support 

Real-time reference services should consider the issue of long-term technical 
support. TalkBack was a commercially available program, but was no longer being 
updated, enhanced, or supported. We were able to use the existing program for a very 
modest cost (almost free), but we were on our own with regard to enhancements and 
technical support. Consequently, we used the program in its original state. TalkNow was 
also free and we could customize the software to meet our specific needs. However, the 
student authors moved on to other projects and are not available to provide further 
enhancements or support. 

It can be extremely valuable for a real-time reference service to hire a full-time 
programmer or have access to a programmer on an as needed basis. Another solution, 
depending on service budget, is to purchase a commercial product that is being updated 
and supported. While “grow your own” software developed in-house by students or 
university staff can sometimes work well (e.g., Gopher and Mosaic programs), it is often 
difficult to maintain. On the other hand, some software companies offer products that use 
the companies’ server rather than the libraries’ server, reducing maintenance and support 
demands. However, these products can be relatively expensive. 

In April 2000, we made the TalkNow software available for anyone to download. 
We hope that others can take the basic code, improve upon it, and then put it out in the 
public domain in a continuing cycle. 

Service Effectiveness 

Our real-time reference service has received hundreds of questions and many 
favorable and constructive comments from users. A more in-depth analysis, using the 
archive of questions, will give us a better understanding of the effectiveness of the 
service. We plan to obtain additional feedback from users through the following 
strategies: 

• A link on the TalkNow page will allow students to submit comments. 

• At the conclusion of a TalkNow transaction, the librarian will ask the user to rate 

his or her satisfaction level. 

• Students who have used the service will be sent a short follow-up questionnaire. 
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Conclusions 



It is vital to have the support of executive management. We found that with the 
University Librarian’s support, the project’s chances of success are improved 
immeasurably. In addition, those implementing and managing real-time reference should 
take certain measures to ensure success: 

• Provide positive reinforcement to the staff members who operate the 
service, and deal with problems as soon as they occur. 

• Give staff discretion to develop their own style and to use their own 
judgment in answering real-time reference questions. 

• Encourage suggestions to improve the service. 

• If possible, phase in the service gradually to give staff a chance to get 
accustomed to the software and the new way of interacting with patrons. 

• Allow staff to “own” part of the project. Make sure they feel invested and 
try to develop an atmosphere that helps everyone to feel involved in the 
service. 

Real-time reference services are in the early stages of development. Chat and 
instant messaging programs represent the beginning of collaborative services that will 
evolve by leaps and bounds during the next few years. TalkNow represents the latest 
refinement in our continuing effort to use new technologies to meet the needs of our 
community. 
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Remote Reference Services at the 
North Carolina State University Libraries 

Eric Anderson, Josh Boyer & Karen Ciccone 

Abstract 

In order to provide equivalent services to patrons in and outside the library, North 
Carolina State University (NCSU) Libraries have implemented a service point within the 
main library for responding to requests from off-site patrons. During their shifts at the 
off-site services desk, staff members answer reference questions received via telephone, 
e-mail, and online chat. 

The new service point has enabled librarians to give their full attention to 
telephone calls. Meanwhile, librarians and patrons at the traditional reference desk are no 
longer distracted by a ringing telephone. 

Staffing the new desk continuously in shifts has dramatically improved our ability 
to answer questions quickly. Response time is considerably less than that during normal 
business hours (approximately two and one-half hours on average). 

This paper discusses the benefits of implementing the service, staffing strategies, 
and some challenges encountered. The Ask a Librarian Web site is located at 
www.lib.ncsu.edu/libref. 



Introduction 

The North Carolina State University Libraries’ Research and Information 
Services Department has implemented a service point within the main library for 
responding to requests from off-site patrons. During their shifts at the off-site services 
desk, staff members answer reference questions received via telephone, e-mail, and 
online chat. 

The off-site services desk reflects the NCSU Libraries’ commitment to providing 
equivalent services to all patrons, whether they come to the reference desk in person, by 
phone, or by e-mail from outside the library. The Association of College and Research 
Libraries (1998) promotes the philosophy that distance learners are “entitled to library 
services and resources equivalent to those provided for students and faculty in traditional 
campus settings.” The Research and Information Services Department has embraced this 
ideal and extended it to all off-campus NCSU-affiliated patrons: faculty members in their 
offices, students doing research in their dorms, disabled patrons who have difficulty 
coming to the library, and others in addition to distance learners. The Research and 
Information Services Department considers the research needs of off-site patrons and 
those of patrons who come into the library to be equally important. 

Off-site Services Desk 

The off-site services desk is located in a small room in the reference area next to 
the traditional reference desk. This proximity is an important factor in keeping staff from 
feeling isolated, allowing consultation with staff at the traditional desk. This has also 
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made it unnecessary to purchase additional copies of reference works. Additional copies 
of telephone books, procedures manuals, library schedules and telephone numbers, and 
other frequently consulted materials have been placed in the room, but the entire 
reference collection is just outside the door. Voice-mail ensures that no call will be 
missed if a staff member needs to step away from the desk for a minute to consult a 
reference source. Staff members also consult a collection of online reference tools 
(currently under development) (www.lib.ncsu.edu/eresources/er_ref.html), which 
includes links to many valuable sources of answers to ready reference questions. 

The physical layout and furnishing of the room continues to improve as desk staff 
suggest ways to make it more functional and comfortable. We have added a non- 
networked computer with Internet Service Provider accounts for diagnosing remote 
access problems. We have also created more desk space and otherwise responded to 
needs as they have arisen. 



Telephone Reference 

One of the major impediments to providing equivalent services to off-site patrons 
was the conflict between answering the telephone and providing on-site patrons with 
undivided and uninterrupted attention. The phone was often viewed as a distraction, with 
desk staff feeling pressured to end conversations quickly. By removing the telephone 
from the reference desk, the new service point for off-site services has enabled desk staff 
to give their full attention to telephone calls. Voice-mail and a “zero out” option (i.e., 
allowing callers to press 0 to reach a human being) ensure that patrons never encounter a 
busy line, and a new toll-free number for distance learners eliminates a potential 
economic barrier to attaining reference assistance. Meanwhile, staff and patrons at the 
traditional reference desk are able to focus completely on in-person reference 
transactions. 

To further improve the quality of our telephone reference transactions, we have 
created written procedures for answering the phone, putting someone on hold, and 
transferring a call. Reference desk training now incorporates a component dedicated to 
remote reference transactions. 



E-mail Reference 

The NCSU Libraries have seen a dramatic increase in the number of e-mail 
reference questions, accompanied by a decrease in the number of in-person reference 
transactions. This is a trend similar to other academic libraries nationwide (Association of 
Research Libraries, 1999). The off-site services desk has helped us to respond to these 
changes. By staffing the desk continuously in shifts, we are able to reply quickly to the 
increasing number of e-mail reference questions. Average response time (including 
evenings, when the desk is not staffed) has dropped from seven and one-half hours in 
October 1999 to four hours in May 2000. Response time during regular business hours is 
considerably less than that (approximately two and one-half hours on average). 

Before the creation of the off-site services desk, e-mail reference questions were 
received in a dedicated library e-mail account using QuickMail and were answered by 
one staff member. The staff were forced to overcome several technical and logistical 
challenges. A shared system for answering these requests was developed, and the e-mail 
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system was converted from QuickMail to Netscape Messenger at the same time. The 
switch to Messenger resulted in the need to create a shared folder that could be accessed 
by all desk staff during their shifts at the off-site services desk. There was also a need to 
ensure that one person (and only one person) would take responsibility for answering 
each question. We solved this problem by having staff only open new e-mail reference 
questions during their assigned shifts at the off-site services desk. If a staff member is 
unable to answer a question during that time, he or she may either pass the question on to 
the next person on the shift or continue answering the question by placing it in a 
subfolder marked “Pending.” Once a question has been answered, it is moved to a 
“Done” subfolder and later deleted. 

We collect statistics on the number of questions received, the time taken for staff 
to respond, and the affiliation of the people asking the questions. We also take note of the 
geographic origin and general subject of the question. 

One of the biggest challenges we currently face is developing a system of quality 
control and staff training for answering e-mail reference questions. Without the 
immediate feedback available with in-person, telephone, and online chat transactions, it is 
sometimes difficult to know how far to go in answering a patron’s e-mail request, or even 
whether a response has addressed the question. We encourage the patron to call the desk 
or reply if his or her question has not been completely answered. We also try to “put a 
smile” on our e-mail responses, providing the same cheerful and attentive customer 
service one would expect at the reference desk. In an effort to improve quality and 
consistency in our answers, we read other people’s responses before they are deleted. We 
have also created a “cut & paste” folder containing exemplary answers to frequently 
asked questions. 



Chat Reference 

NCSU Libraries decided to implement a chat service in order to provide patrons 
with immediate response and to allow patrons with one phone line to continue online 
searching while asking a reference question. 

Initially, we investigated two software solutions for our chat service: America 
Online’s Instant Messenger and a free campus IRC chat server. Both options had 
limitations. Instant Messenger required patrons to download and configure the software; 
the IRC server required us to set up an appointment system to indicate when a patron 
wants to chat and required patrons to log in with their NC State IDs. 

In order to minimize the effort on the part of the patron, we decided to pursue chat 
software that was much simpler for patrons to use. For instance, the chat service offered 
by the Lippincott Library at the University of Pennsylvania 

(www.library.upenn.edu/resources/reference/business/ref-business.html7business) uses 
LivePerson chat software. This system allows patrons to reach the chat service by simply 
clicking on a “Live Reference” icon. It was clear that we would have to spend money on 
software. LivePerson and similar products cost hundreds of dollars a month, but our free 
options came with too many limitations. 

We selected LSSI’s Virtual Reference Desk, a product that offers a chat interface 
as well as a collaborative Web-browsing feature (i.e., the patron can view what is on the 
librarian’s browser and vice versa). Reference librarians can point a patron’s browser to 
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the library’s catalog and show her how to use it, and the patron can show the librarian 
where she was looking for information when she asked for help. The patron and librarian 
can talk to each other on the phone or by chatting. This requires nothing from the patron 
but a browser - no downloads, no plugins, no configuring, no IDs, no appointments. We 
plan to implement the service in January 2001. 

We are excited about collaborative browsing for several reasons: 

1 . During a chat session, librarians can show patrons how to find something on the 
Web, instead of just telling them. The Web is a visual and interactive medium; 
writing about it in a chat screen is like describing how to a paint a picture. 

2. Librarians will use the collaborative browsing feature with patrons who contact us 
via chat as well as those who initially contact the library through phone and e- 
mail. In other words, whenever we want to demonstrate how to use something on 
the Web to a patron who is not at the reference desk, we can just ask them to go to 
our Ask A Librarian Web site and click the Live Help button. 

3. NC State has 1,400 distance education students. One way to show them how to 
use the library’s Web site (including the catalog, databases and e-joumals) is 
through Web co-navigation. This can be done either one-on-one or with a class at 
a remote site. 

4. Collaborative software represents one of the most interesting frontiers on the 
Web. Commercial sites like Lands’ End allow customers to shop together. 
Business people use software such as NetMeeting and WebEx to hold meetings 
online. As a library, we want to experiment with this technology and see where it 
leads. 



Web Site 

The Ask a Librarian Web site includes links to information and assistance 
(www.lib.ncsu.edu/librefQ, and includes our policy for responding to questions from non- 
NCSU patrons: “This service is for students, faculty, staff, and other affiliates of NC 
State University. If you are not affiliated with NC State, we regret we can only answer 
questions about services and collections unique to the university. Other inquiries may be 
referred to more appropriate libraries or sources of information.” 

This notice is a necessary response to the high volume of questions we 
receive. However, we attempt to assist non-NCSU patrons whenever possible, 
including referring them to local resources they might be able to utilize. 

Also included on the Web site are examples of the types of questions that 
can be asked and a link to our FAQ. The site also includes links to options for 
telephone, e-mail, and online chat reference. 

Future Directions 

As the off-site services desk matures, we some requests for improvement. We 
would like to upgrade our method of gathering statistics in order to assign responsibility 
for a question to a specific librarian, automatically track and record response time and 
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other statistics, and save selected, edited responses in a database. We are exploring 
various applications, including Remedy, as possible answers to this need. 

Additionally, we wish to increase the amount of peer review in e-mail, phone and 
chat reference. At a traditional reference desk, librarians listen to each other answer 
questions and learn from each other’s best practices. Phone, e-mail and chat usually limit 
the librarian and patron to one-on-one communication, making the sharing of reference 
interviews more difficult. We plan to explore ways to help reference librarians share their 
online successes. 
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The Virtual Reference Desk Network 



Blythe Bennett 

Information Institute of Syracuse 

Presentation 

Introduction 

The Virtual Reference Desk Network accepts out-of-scope and overflow 
questions and routes them to more appropriate services or volunteer librarians for 
response. The collaborative network began in January 2000 and continues to add new 
volunteers and services to assist patrons, provide professional development for 
volunteers, and support AskA services for a win-win-win situation. 
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Virtual Reference Desk Network 



Seattle 

October 16, 2000 



VRD Project 



^Clearinghouse 

EDigital Reference Conference 
^Publications 
ElAskA Consortium 
Elnstruction and Support 
EResearch and Development 



VRD Project 



^Services 

^Collaborative AskA Network 
ELearning Center 
EAskA+ Locator 
EIncubator Software 



Project Goals 



^Connect K-12 education community and 
beyond to experts and expertise. 

38 Provide resources and support to AskA 
services in all contexts. 

38Set standards for digital reference. 
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VRD Sponsors 




Service Types 


8€U.S. Department of Education 




3€General information referral 


National Library of Education 




[^Internet Public Library 


3€With support from the White House Office 




SLibrary-based services (e.g., Ask A Librarian) 


of Science & Technology Policy 




^Specific subject area 
QAskERIC 

E3MAD Scientist Network 






[^National Museum of American Art Reference 
Desk 



How Do Organizations 
Become Involved? 



8€Volunteer with an AskA service managed 
by the organization. 

8€Volunteer with a preexisting AskA service: 
0MAD Scientist, Dr. Math 
8€Volunteer with VRD as subject expert. 



How Does VRD Help Start A 
Service? 



§€ Provide Web based AskA software 
(Incubator). 

Provide server space on the VRD site. 
8€Provide training for administrators and 
trainers. 

'g.AskA Starter Kit. 
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Collaborative AskA Service 



$ Launched January 2000 

018 Services 

056 Volunteer Librarians (on call) 

01.25 Staff 

0approx. 300 questions/month 
0over 30 countries 

& Seeking new services/individuals to join network 
0 Library- related 
0 Subject-specific 
Online training 



VRD Network 2000 
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VRD Network Members 




VRD Network Members 


§€ Science/Math related AskA services: 

nri TKn MAPI MflKi»Ar4/ 




96 Non-Science related AskA services: 


Led J ne riAU bcienost neiworK 
0 Ask Dr. Math 




ElAskERIC 


0 Ask Dr. Universe 




13 Kentucky Center for School Safety 


12) Ask Shamu 




BDigital Ubrary and Archives at Virginia Tech. 


(21 Dlno Russ's Lair 




^Internet Public Ubrary 


El Science Une (in UK) 




ElMorris County Public Library (NJ) 


0 Eisenhower National Clearinghouse for Mathematics and Science 




BAsk Joan of Art, National Museum of American Art 


Education 




(Smithsonian) 


□ Teacher 2 Teacher (from Math Forum) 




23 Library of Congress's American Memory 


0 Ask Jake the Sea Dog 






0 Ask a Space Scientist (NASA) 




9§Network Information Specialists: 


0 Environmental Protection Agency 




(350+ trained librarians (school, public, academic) 






(3 MLS students (some interns) 
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VRD Volunteers 




Network Triage Process 


Elementary school librarians * 13 




^Out-of-scope and overflow questions from 


& Middle school librarians - 9 




AskA services sent to VRD 


SSHigh school librarians - 13 




9§VRD re-routes to more appropriate 


& Multi-age school librarians - 8 




service, or question is answered by VRD 


^Academic librarians - 7 




volunteer 


& Public librarians - 2 




3€If answered by a volunteer, copy of 


&MLS students - 4 




response is sent to archive 


SSPart time paid staff - 1 







Triage Examples 



S§AskERIC: origin of French people in 
Louisiana 
0VRD volunteer 

&MAD Scientist Network: teaching atomic 
theory 
E3ENC 

&IPL: curriculum design 
ElAskERIC 



Sample K-12 Questions 



% What is the exact length of the planet Mars' day 
to 4 decimal places? 

ft I am doing a paper on what I want to be when I 
grow up and I've chosen teaching but I don't 
know how to find the information I need please 
help!!! 

U Where can i find pictures of the poverty and 
destruction during the Great Depression? 
i am looking for information on domestic violence 
to write my term paper.could you please e mail 
me any information and web pages that you 
have. 




4 



115 



Post Secondary Questions 



3£ A student wanted to know if it is true daddy 
long leg spiders can kill black widow spiders. I 
would like to know more about this subject so I 
can provide a correct response. 

3^ Why is the "speed of light" (c) so critical in 
converting energy into mass or mass into 
energy? Both a quantitative (mathematical) and 
qualitative explanation would be helpful. 




Science Subtopics 




Quirky Questions 



3€I am investigating the surface area of a camel's 
hoof and its relationship with the weight being 
loaded. 

3€ what is the average air velocity of a European 
swallow? 

3€ what is the perfect bra? 

3SWe were wondering how much the average cow 
craps in one day. 
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Out of Scope 



How long does it take blood to dry? Given these facts: 
Temperature: 90-95 degrees, blood is on concrete, time of 
day: 6:00 p.m. - 7:00 p.m. in July, which is usually a hot and dry 
month. Please don't think I am weird. My husband is a former 
homicide detective and is in the process of being sued. This 
involves a homicide he investigated. 



VRD Network Policies 




Advice 




Controversial topics 




Copyright 




Inappropriate comments/questions 




§€ Interlibrary Loan (ILL) 




§€ Liability 




& Privacy 




Selection Policy 





Thank You Notes 



I just wanted to thank you for taking the time to 
respond to my questions. Your response was full 
of wonderful information; I am pleased to know. 
Keep up the good work. 

3€Thanks for your advice (and your brain) It will 
help me allot with my assignment, and I think 
(and hope) it will impress my Science teacher, 
but it is very hard to impress "smelly Reynolds" 



Benefits 



3€for AskA services 

E3 Forward out of scope questions to VRD to be answered 
by appropriate services or experts. 

El Forward some overflow questions to VRD to be handled 
by VRD Information specialists. 

0Assist with shaping standards in online Q and A. 

EJVRD Consortium membership 
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Benefits 



Pfor patrons 

23 Questions are forwarded to the most appropriate 
experts. 

El Reliable service: All questions receive responses from 
subject specialists or general information specialists. 

23 Seamless transfer of original question, resulting in less 
waiting time 



Some issues 




Pliability 




Presponse rate 




Padvice 




Pscope - topics and age range 




Parchiving responses 





Future of VRD 




VRD Learning Center 


P Launch of Learning Center 
P Launch of incubator service 
PPublic access to collaborative AskA service 
PCross-service knowledge base 
PSoftware development for internal 
processes 

PRecruit new services and experts 
SMatchmaking database 
PContinue dialogue in digital reference field 




PPublic section for patrons 
EDAskA Services 
EDPreviously-asked questions 
SFAQs 
STop sites 

P Parents & Educators 
PResearch Skills 
P Search 





Virtual Reference Desk is... 






Creating foundations for a national. 




Blythe Bennett 


distributed digital reference service to: 




Learning Center Coordinator 


SConnect the K-12 community and beyond to 




Virtual Reference Desk Project 


experts through the Internet. 




Information Institute of Syracuse 


[^Provide resources and support to "AskA" 






services. 




blytheb@vrd.org 


0Set standards for digital reference. 




http:/ 7 w w w . vrd . o rii 
800-464-9107 


http://www.vrd.org 
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Transitioning from Call Center to Digital Reference Service 

Joe Schumacher, Roseanne Schwartz, and Kit Pitkin 
CIESIN at Columbia University 

Presentation 

Introduction 

Columbia University's Center for International Earth Science Information 
Network (CIESIN) is a research and data center concerned with human interactions with 
the environment. As such, CIESIN has two projects that provide product support and 
reference assistance through its user services office. User services at CIESIN began as a 
product (data and services) support center, much like a commercial call center, as most 
user requests arrived by phone and were technical in nature. With the advent of the Web 
and the evolving nature of CIESIN’s projects, CIESIN now provides digital reference 
services in addition to fulfilling responsibilities to support our products. This paper 
describes the changing responsibilities and the actions taken to meet user needs. 

The two projects at CIESIN that have a user services function are the US Global 
Change Research Information Office (GCRIO) and the Socioeconomic Data and 
Applications Center (SEDAC). CIESIN’s user services office operates the Ask Dr. 

Global Change reference service within GCRIO and offers product support and limited 
reference service in SEDAC. 

After a self-analysis of our services, several steps were taken to incorporate 
digital reference principles into existing operations. We have redesigned many of our 
Web pages to guide and invite user inquiries, rather than just let questions “happen.” By 
guiding the questions (clearly stating our area of expertise, types of questions we will 
entertain, set user expectations, etc.), we hoped to reduce frequently-asked, overly 
general, out-of-scope and prank questions. We also wanted to improve question quality, 
eliminate the need for user services staff to act as an intermediary between the user and 
our science staff, and to move away from e-mail as a prime interface. 

An important part of our new approach was to integrate help desk software into 
our services. We purchased RightNow Web from RightNow Technologies. By 
developing a database of questions and answers, we used the help desk software to 
encourage users to browse frequently-asked questions (FAQs) and conduct keyword 
searches before posing their questions to user services staff. The help desk gives us much 
better ability to track questions and produces an excellent array of reports. 

The Ask Dr. Global Change service went online on 1 April 2000, using the help 
desk software and our newly designed interface. Based on the first six months of 
operation, we have seen an approximate 50% reduction in the number of questions asked. 
We believe the reason for this is that many users are finding answers to their questions by 
browsing or searching the archive of questions and answers. There is roughly a 6: 1 ratio 
between FAQs browsed and questions asked and keyword searches and questions asked. 
People are finding the answers to their questions almost immediately and without having 
to pose those questions to user services. 
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A Management Model for Digital Reference 
Services In Large Institutions 

Barbara MacAdam 
Suzanne Gray 

Abstract 

Large, decentralized information organizations face a unique set of challenges in 
effective management of virtual reference service. Proposed here is a model that is 
particularly relevant to academic libraries in large research institutions, but extendable to 
other types of organizations with similar characteristics. The five critical issues examined 
include: integrating virtual reference service with existing services; allocating fixed 
resources; acting as an effective advocate to secure organizational support; developing a 
distributed service model integrating specialized, subject-domain expertise; and targeting 
and serving disparate segments of the user community. These concepts and strategies will 
be most useful to managers in information organizations who have responsibility for 
reference service development and need solutions designed specifically to achieve service 
goals for virtual reference by working within and across the larger organization. 

Introduction 

Information professionals who made the assumption that virtual reference services 
development would share most of the challenges of traditional reference models, and 
roughly the same solution strategies, have quickly realized the obvious and significant 
differences. Moreover, large, decentralized information organizations face a unique set of 
challenges in effective management of virtual reference services. A sound management 
model applied to academic libraries in large, research institutions may have useful 
application to other types of organizations with similar characteristics. The factors unique 
to the larger organization can be neutral elements, barriers, or positive agents in any 
effort to develop and manage digital information services, depending on the ability to 
manipulate them successfully in a new initiative. 

Characteristics 

1 . A broad mission - A typical academic library mission statement might be to support 
the research and educational endeavor of the campus, to serve the larger community for 
the public good, and to acquire and preserve the collection for the knowledge base of the 
future. While almost any activity can be justified under this mission, it is harder for any 
particular service initiative to demonstrate mission critical importance. 

2. Various and sometimes conflicting goals - Conflicting goals might include: to 
provide timely, location-independent information assistance at the time of need; to foster 
information literate undergraduates; to serve as research partners with the teaching 
faculty; to maintain welcoming and widely accessible physical facilities; to sustain the 
culture of the book; and to foster the adoption of new information technology. Tensions 



occur when large organizations try to turn fundamental values into operational strategies. 
Digital reference service must advance some clear goal to be perceived as though it is 
worth the resources and effort allotted; these resources and services will have been 
inevitably diverted from other important initiatives. 

3. Decentralization - The work of the organization is spread over multiple units, 
separated by function (e.g., technical services, public services, systems, financial and 
personnel operations) and often isolated geographically and administratively. Large 
public service operations may have multiple physical service points, multiple reference 
departments, highly differentiated resources, variations in policy and process, and even 
unit-specific mission statements and goals. Virtual reference service, by its very nature, is 
at odds with this organizational segmentation, yet its success will depend on the 
manager’s ability to work within existing realities in a distributed service model. 

4. Centralization - The largest and most distributed organizations have centralized some 
key operations including systems support, software approval, purchasing, or operating 
budget. This can potentially hinder the manager’s autonomy in making service decisions, 
establishing priorities, and controlling resources. 

5. Widely dispersed and over-committed resources - Large organizations have large 
budgets and a large staff, but these are often deployed broadly and thinly across the 
distributed service operation. It is critical to understand the dimensions of 
decentralization versus centralization and build cooperation in order to develop 
effectively the critical mass of resources necessary to sustain a digital reference service. 

6. A non-homogeneous user community - Faculty, graduate students, undergraduate 
students, alumni, the state or regional community, a global research community, 
corporate partners, and consortial agreements may all place varying demands on a virtual 
reference service. Although their needs vary, along with the information organization’s 
perceived obligation to meet those needs, virtual reference users will inevitably represent 
a broader spectrum of the larger community than traditionally served. One of the most 
critical challenges is designing a central service gateway, balanced with a policy and 
operational structure that preserves the desired service focus for segments of the user 
community. 

7. An established set of services - Services are highly entrenched with a history of 
budget support, service structures, staffing, and demonstrated importance to the user 
community. It is unrealistic to expect existing services (and their advocates) to just move 
over or relinquish energy and resources to digital reference development. 

8. Highly-specialized subject expertise - Particularly in research libraries, the service 
structure depends heavily upon the research consultation role of subject specialists spread 
across the organization. This includes specialists of discipline-related knowledge and 
advanced technical expertise. Unless goals for the service are merely to provide basic 
information assistance, a successful model for digital reference must include ways to 
preserve and foster the links between users and the specialized knowledge providers. 
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9. A complex institutional and organizational culture - Large organizations present an 
often mystifying array of traditions, unwritten rules, territorial issues, protocols, style, 
and implicit values that hinder a clear understanding of the most effective methods for 
achieving goals. Virtual reference service grows from an environment that is dynamic, 
entrepreneurial, democratic, and inherently impatient. Sustaining the development and 
growth of a service, however, requires stability, support, and resources that ultimately 
only a broad-based, firmly anchored position in the organization can ensure. 

10. Historically entrenched user expectations - Virtual reference may be perceived by 
some as a non-essential program draining resources, time, and effort away from the work 
of more traditional reference service. Faculty may deplore efforts that permit students to 
stay out of the library physically, and administrators may worry that staff, already hard- 
pressed to meet needs of primary users, are now responding to demands from users 
around the world. 

11. Complacency, conservatism, and inertia - Large organizations are generally secure 
in their traditional success and complacent about their importance to the user community. 
The sheer volume of services and activity on a day-to-day basis can overwhelm efforts to 
identify emerging user needs and to respond in a timely way. 

12. Process - Large organizations have a process for everything, ensuring that there is a 
way to get everything done, and guaranteeing that major service innovations run the risk 
of being over scrutinized. 

13. Pride, creativity, know-how, vision, energy, dynamism, resources - All of the 

above elements reside somewhere in a large organization, especially today’s research 
library. It is critical to marshal these qualities with the same urgency with which one 
would identify and deploy more tangible resources. 

Turning Critical Issues into Achievable Goals 

Defining an Appropriate Role and Integrating Virtual Reference Service with Existing 

Services 

1 . Identifying service gaps - Precisely what user needs are currently unmet? If there is a 
lack of reference service across long hours in small, divisional library units, or support is 
needed for users working primarily with digital resources, it needs to be made explicit 
why the best solution is not merely expanding existing information services. 

2. Confronting changing user patterns - Has reference desk traffic gone down? More 
than likely, it has, since students now have 24 hour access to the Web through the campus 
computer labs or residence halls, and many faculty have developed a full curriculum in 
the Web environment, sustaining relationships with and among their students in that 
environment. 
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3. Establishing a shared purpose and clear priorities - The goal is not to create a 
digital reference desk. Presumably, the goal is to provide outstanding information 
services for the user community, and digital reference service is a strategy that is 
responsive to unmet user needs. This is the foundation that will ensure strong connections 
to present services. 

4. Laying an existing service element to rest - What is the organization willing to give 
up? Put another way, somewhere in the array of information services lurk components 
that users neither need nor desire; they continue because staff is unwilling to relinquish 
them. 

5. Combining seasoned experience with creative innovation - Developing projects in 
the digital environment takes knowledge of technology, creativity, and a zest for 
experimentation. Sustaining a working service requires experience, stability, dedication, 
and sound management. Changes in the information environment and professional 
education have been so profound and happened so quickly that the potential exists in 
most larger organizations to involve veteran and newer professionals whose synergy will 
ensure successful digital service development. 

6. Applying or adapting existing policies - Existing policy typically includes 
safeguards for privacy of patron transactions, levels of privilege for different segments of 
the user community, and guidelines for referrals. Policy is designed to make the majority 
of transactions flow smoothly, consistently, and predictably from the user’s point of view 
and to ensure that users are enfranchised appropriately and treated ethically and that 
organizational resources are expended efficiently. The degree to which virtual reference 
services integrate successfully with existing services is determined somewhat by the 
degree to which users neither lose nor gain status in the digital environment and to which 
existing policies remain in new services. Inevitably, there will be areas, internal and 
external, where existing policy will be insufficient. New policies should be consistent in 
values and outcomes of existing policy. 

Allocating Fixed Resources Effectively 

1. Administrative authority and budget flexibility - Virtual reference services must be 
sustainable, and in most organizations that means redirecting resources from an existing 
budget. During an entrepreneurial or pilot phase, one-time money for capital equipment, 
temporary staffing (e.g., interns, residents, or use of lapsed salary money), or seed/grant 
money from inside or outside the institution are all reasonable ways to get an initiative 
off the ground. A viable, long-term service relies on diverting base budget resources; this 
means planning, which includes working within established processes for equipment 
requests, reassigning existing staff, or shifting money among personnel lines. The digital 
reference service manager needs authority to allocate resources or the support of an 
administrator who does, along with the experience and confidence to make aggressive 
budget decisions. 
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2. Staffing - In the larger information organization, it is not realistic to build a service 
solely on the assumption of hiring additional staff, exclusively devoted to providing 
digital reference. The best staff model integrates existing staff to the appropriate degree, 
with enough net staffing increase to make the service viable. In the search for additional 
resources, late-night reference staffing might be structured to handle virtual reference 
queries in slow periods between patrons; staff members who traditionally have not 
worked evenings or weekends might be willing to provide service from home over 
extended hours. Digital information services in the large, decentralized organization must 
invariably migrate toward a distributed staffing model. 

3. Collaboration - Fostering cooperation among individuals and units within and outside 
the library serves two purposes. It can produce a critical mass of staff, expertise, and 
other resources lacking in any individual unit. It can introduce the combined elements of 
flexibility and stability for service development. 

Being an Effective Advocate and Ensuring Administrative, Staff and Technical Support 

1 . Gaining staff buy-in - The quality and longevity of a digital reference service depends 
on the staff who provide it. It is critical that staff share a vision for information service 
and perception of unmet user needs. They should be involved in every step of service 
planning, implementation, and operation. Managers should demonstrate that they are 
conscious of the very real additional demands of digital reference. One should be clear as 
to where resources are coming from, and convey that thought is being given to long-term 
needs: from resources, to protecting existing service, to staff training. It should be made 
clear that staff will not give up their direct role with users; digital reference service can 
actually foster contact with their user community. The enthusiastic manager also needs to 
be realistic as to how quickly service development can occur and to make sure 
administrative zeal doesn’t outpace the time and energy of committed but overworked 
staff. 

2. Resourcefulness, independence and risk-taking - Ideally, every information service 
manager would find the information technology infrastructure, administrative support, 
resources, and expertise readily available to develop and sustain a digital reference 
service. Because larger organizations are usually running at full speed with staff and 
other resources fully committed, the wise manager finds ways to implement digital 
reference service using a low-key approach. The next critical phase of operations is to 
elicit higher administrative approval based on a demonstrated proof-of-concept. The 
manager who avoids requesting additional resources for every new service initiative will 
also avoid the implicit request for prior approval and the accompanying scrutiny for an 
endeavor in its early days. 

3. Determining useful allies including partners outside the organization - It is 

essential to build a cohort of colleagues who share a vision, understanding, and 
enthusiasm for digital reference. Staff at any level of the organization can be helpful in 
many ways. Overlapping goals create strong partnerships. There may be programs 
underway across campus, or within the library, where digital reference service fills an 



identified unmet need. Community service programs, relations with alumni, distance 
education, public relations, admissions recruiting, technology support/help-desk 
initiatives or other information services, are all examples of opportunities for cooperation 
which may produce anything from resources to critical affirmation of shared institutional 
values. 

4. Clear goals and persuasive communication - In spite of the persistent belief that a 
good idea will sell itself, there is no substitute for a clear, concise statement of the goals, 
audience, and projected benefits to the user community for a virtual reference service. 
Because it is new, requires considerable development of the service underpinnings 
(software/network support, staffing, policy), and suffers from the incomprehensible but 
pervasive belief that digital content fosters completely independent users, digital 
reference service requires a justification where traditional services go unchallenged, 
despite declining statistics. 

5. Demonstrable success - Evidence of success can include steady (or exponential) 
growth in use, proof that the service is reaching a primary user community, and degree of 
user satisfaction. These will help illustrate that the service is worth the allocation of 
resources, effort, and attention required. Any successful new endeavor generates a buzz 
of good will; administrators will value and affirm a service with demonstrable significant 
outcome for the campus, colleagues will want to participate, and the user base will grow 
by word of mouth. 

6. Evaluation - The first obligation in managing a digital reference service is to provide 
outstanding service to users. Data, including patterns of use by targeted segments of the 
primary user community, have value in providing management information for ongoing 
service development and improvement. The second obligation is to provide tangible 
evidence that the virtual reference service is meeting a critical need, thus justifying the 
continued allocation of resources. The potential for research is enormous in this area, 
with a wealth of data waiting to be tapped. However, the manager who honors the first 
two obligations will ensure the success and viability of a digital reference service and be 
in a position to engage in the latter activity. 

Establishing a User-Centered Service Model that Balances Centralization and 

Decentralization 

1. One-stop shopping - Users need a central gateway to digital reference service. One 
option is to offer a button labeled “Ask Us” or “Have a Question?” on the organization’s 
Web site. This could help consolidate existing e-mail boxes and replace a variety of Web 
contact points. In a highly distributed organization, the central representation of digital 
reference services can be a significant opportunity to create a visible, easy to use, and 
consistent information service point for users. 

2. Integrating subject-domain expertise - A central digital reference service does not, 
and should not, supplant the relationship between library subject specialists and the 
faculty and students they serve. The virtual reference desk can operate as a clearinghouse 
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and referral point for users who do not know whom to contact, and actually create new 
links between librarians and users. Faculty who might normally go directly to a particular 
librarian for discipline-specific consultation may prefer to use a central service for more 
general questions or when the staff member is not available. There needs to be agreement 
across the organization that a significant amount of reference activity will continue to 
transpire directly between librarians and users, but that subject specialists will anticipate 
and be prepared to respond to inquiries through the central service. The software/network 
environment should support distributed operations. 

3. Referral structure - Referrals present a different set of problems in digital reference. 
Because many virtual reference services make a commitment to answering questions 
within a set time frame, monitoring the response time is critical. In a distributed 
environment, those specialists within the organization who are not participating directly 
in the centralized reference service may have different goals and standards for their 
service. Once the question is forwarded to another service point, the centralized service 
loses control over the quality of the answer and the response time. 

This problem can be dealt with on two levels. The first is to gain an understanding of the 
service policies for all of the specialized virtual reference service points in the 
organization. Once the policies are understood, there may be room for negotiating 
common standards of service. However, if the specialized library is unwilling to handle 
requests from those outside their primary clientele, the centralized service may need to 
avoid referring these requests to that service point. A second strategy for handling 
referrals is to clearly notify the user that the query has been turned over to the specialized 
library. Providing a Web address for the specialized service point allows the user to 
follow up on the inquiry if a timely response is not forthcoming. 

4. Setting and achieving consistent response goals - Users expect a consistent, 
predictable turn-around time for the questions they submit. Their expectations are often 
colored by the possibility of immediate access of content allowed in the Web 
environment. Typical response goals might include immediate confirmation of inquiry 
(system generated), a response or initial brief response (for more complex questions), or a 
message that the question is being referred, all within 24 hours, with a target of final 
response within one week for all questions. The best practice is to pick the shortest but 
most realistic response time and clearly communicate this to the user. Otherwise, users 
may be discouraged by a long wait or disappointed when the service fails to respond 
within the stated timeframe. Citing alternatives on the Web site (e.g., calling the 
reference desk for urgent, quick answers) will help meet user expectations. 

5. Staff training and development - Beyond reference expertise, staff participating in 
virtual reference service need to have a clear understanding of the software/information 
technology being used to support operations, and the specific goals, policies and 
guidelines for service. Within this framework, they will make judgments in constructing 
an appropriate response. As more staff and units across the larger organization participate 
in the service, it is important to build a shared understanding of service goals. Staff may 
find they prepare long responses in the virtual environment. Benchmarks are useful to 



define how long an average response should take before a referral or other action occurs, 
and models of well-crafted responses can help even experienced reference librarians 
develop communication skills in an electronic mode. 

6. Quality control - The most fundamental way to ensure that digital reference service is 
consistently excellent is to define realistic service goals, accompanied by workable 
policies and procedures, with participating staff fully cognizant and invested in them. A 
digital reference service in a large research library must be predicated on the assumption 
that reference staff are expected to work with autonomy and to tailor responses to the 
specific situation. A well-designed Web form can elicit enough information from users to 
help offset the lack of a true reference interview. Reference styles vary among 
individuals. Staff in an undergraduate library, for example, might reasonably be expected 
to construct answers that are more instructional in nature when responding to students, 
while medical library specialists emphasize speedy, complete information to their faculty. 
Monitoring activity periodically and offering models for good responses, along with a set 
of standard answers for common requests are also effective strategies. 

Targeting Services to the Key Segments of the User Community 

1 . Needs analysis - What unmet information service needs do the various segments of the 
user community have? The first step in targeting services is to clearly define the primary 
clientele and the institutional obligations to a wider community. Undergraduates may 
need real-time, immediate assistance when working from a residence hall or lab in the 
electronic environment. Faculty on sabbatical or doctoral students at a remote research 
site, for example, have other needs. 

2. Tiered services - Because a wide range of users are attracted to a digital reference 
service, traffic from non-primary clientele needs to be controlled. Often the Web site of a 
large research library attracts non-affiliated visitors from around the world whose 
questions may not be directly related to the institution’s collection or mission. Closing the 
service off to those from the outside is one approach; however, this is often contrary to 
institutional policy regarding telephone and face-to-face reference service, which usually 
does not require proof of affiliation from users. Barriers such as time, distance, or phone 
charges minimize the number of non-affiliated users of traditional reference services, but 
virtual reference services break down these barriers by providing easy, often low cost 
access to the service. 

At most institutions, resources allocated to virtual reference services will not allow staff 
to fully meet the needs of non-primary clientele. Tiered service can create barriers for 
non-primary users while providing easy access to primary clientele. Services can be 
tiered based on response time, the amount of information requested of the user, and the 
quality of the answer provided. For affiliated users, the goal is to continue to decrease the 
response time in order to make the service as convenient as possible. For non-primary 
clientele, a more significant wait, perhaps of a week or more, may encourage them to 
satisfy their information need through another service. The use of separate forms for 
different user groups can also create barriers to non-primary clientele. Requesting 
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additional information from these users helps to provide a clearer context for answering, 
while also requiring additional effort on the user’s part to generate a query. Additionally, 
primary clientele are provided with an in depth response that is frequently instructional in 
nature, while non-primary clientele may be provided with a few Web sites or a referral to 
a specific search engine. In order for these barriers to be effective, this policy must be 
clearly communicated to users on the virtual reference Web site. Additionally, staff must 
be trained on the service goals for each category of user to maximize efficiency. 

Tiered service allows for better allocation of staff time to handle incoming queries. 
Different types of staff can be utilized to handle the different categories of requests 
generated by a tiered service. 



Current Challenges 

The Perfect Software 

A still elusive goal for large-scale information services is the complete and seamless 
integration of digital reference operations. The ideal management software would support 
Web-based asynchronous and real-time interaction and telephone and site-based, face-to- 
face reference; facilitate the exchange of digital content; provide flexibility to partition or 
centralize the environment as needed in a distributed service organization; and provide 
the tracking, archiving, search capability, and use-report capabilities critical for the 
effective management of ongoing operations. 

Intellectual Freedom, Privacy, and Access 

The digital reference environment turns each of these issues into a double-edged sword. 
Users may communicate anonymously with greater ease and may engage in more 
egregiously offensive or inappropriate behavior. At the same time it may be more 
difficult to protect an individual’s privacy when detailed information may be captured in 
the initial Web form, and queries are referred or archived. Network authentication, or 
licensing agreements, may present unexpected barriers to service access. On the other 
hand, the accessibility and visibility of the organization’s Web site will expand the user 
community, requiring hard decisions about balancing the public good with the need to 
serve primary clientele. 



Evaluation 

Virtual reference service will invariably grow even in the absence of aggressive 
promotion, but the real challenge is to demonstrate effective and high-demand service for 
the primary user community. Collecting detailed use and user statistics depends on 
adequate data capture and reporting capability within the system, but may present another 
set of confidentiality issues. The more critical need is to develop the goals, service 
performance measures, and data analysis methods for meaningful, systematic assessment 
where the range of complex activity is distributed across the organization. 
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Expanding Services 



Continual identification of unmet user needs will provoke considerations of new 
dimensions of service. For example, a real-time chat component may offer a solution for 
undergraduates who often cannot wait for virtual reference services that take one or two 
days to respond. Controlled expansion, targeted to primary user groups’ needs, will result 
in clearer strategies to reallocate resources appropriately. 

Cooperation and Competition 

Cross-institutional service collaborations, commercial information service development, 
and the development of tools and user interface designs that foster independent use of 
digital content all have enormous potential to change the virtual reference landscape. 
Large, research institutions face a unique challenge to provide substantive, highly 
specialized, discipline-related information and research assistance to a broad user 
community. Academic communities are also dependent on successful relationships 
among units and individuals working within an institutional culture. Wise management of 
digital reference services should be able to identify clearly where needs of the user 
community can be better served through cooperative initiatives or broad-purpose, 
commercial information systems. 
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Transitioning from Call Center to 
Digital Reference Service 

Joe Schumacher 
Roseanne Schwartz 
Kit Pitkin 

CIESIN at Columbia University 
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CIESIN User Services 

• CIESIN - Center for International Earth 
Science Information Network at 
Columbia University 

• Composed of several funded projects - 
two have user services component: 

- GCRIO 
- SEDAC 
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CIESIN User Services 

• Began in 1993 as product (data and 
services) support center 

• Anticipated that most inquires would be 
by phone (and many of them) 

- In reality, almost all requests are via e-mail 



CIESIN User Services 

• Staffed by: 

■ Subject area experts 

■ Tech support and customer service 
representatives - no longer 

■ Reference librarian added later 

• User Services Office also has outreach 
and reporting functions 
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GCRIO User Services 

• GCRIO - US Global Change Research 
Information Office 
~ Sponsored by USGCRP 
- Disseminates information about global 
environmental change 
-Operates "Ask Dr. Global Change" service 
• Have moved in past year to a more efficient 
service based on help desk software 
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SEDAC User Services 

• SEDAC - Socioeconomic Data and 
Applications Center 

- One of eight NASA earth science data centers 

- Archives and distributes data on "human 
interactions in environment" 

- User Services Office 

■ Supports data products and services 

■ Offers limited reference support 

■ Coordinates with other data centers through User 
Services Working Group (USWG) 
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Issues 

• General 

- Originally intended to be call center 

■ Institutional memory difficult to change 

• Project specific 

- GCRIO - Dr. Global Change 

■ Questions invited 

• Answers vs. Reference 

- SEDAC 

■ Questions "happen" ~100/month 

■ User services often acting as intermediary 

■ Need to provide product support and reference se rvice 

VPD2000 Cl ESIN bp> • [.cW Wth penanco »c<n C1ES1N. liumly 



Service Assessment 

• Four types of questions that we wished 
to reduce: 

- Frequently asked 
-Overly general 

- Out-of-scope 

- Prank 

• Had little information about user 

• Difficult to track question 
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Goals for Improving Service 

• Become more efficient 
- Receive fewer questions 
-Offer quicker response 

• Improve question quality 

• Eliminate need to act as intermediary 

• Move away from e-mail as prime 
interface 




Becoming More Efficient 

• Provide context for questions 

• Frequently asked questions 

- Improve Web site to avoid questions in the first 
place 

- Provide database of questions and answers that 
can be browsed and searched 

• Overly general questions 

- Set expectations 

- Provide links/ references to authoritative resources 
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Becoming More Efficient 

• Out-of-scope questions 

-Clearly state types of questions we will 
entertain and types we will not 
• Reiterate, reinforce, restate subjects for which 
we offer assistance 

• Prank questions 

- Design interface to guide questions rather 
than ask for any, open-ended question 




Implementing Help Desk Software 

• Chose RightNow Web from RightNow 
Technologies 

- http://www.rightnowtechnologies.com/ 

• Interface to a database of questions 
and answers designed to: 

1. Encourage FAQ browsing 

2 . Allow keyword searching of FAQs 

3. Ask a question 
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RightNow Web Features 

• FAQs rated by users 

- "Best" rise to top 

• Can assign questions to categories and sub- 
categories 

• Question status and tracking 

- New, assigned, waiting user action, solved 

- Private, proposed, public 

- Can view question history 

• Excellent array of reports 




Number of Dr. Global Change Requests by Month 







Help desk software installed 1 April 2000 
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Dr. Global Change Requests per 1000 Unique 
Users 



B 3 

1 3 
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Dr. Global Change mentioned in Science and Physics 
Today in July and August 2000 



• Doubling of unique users during June 2000 with 
release of National Assessment draft report 







Preliminary Assessment 

• Initial populating of Q and A database not trivial 

- Significant editing required 

- Manager's need for huge database 

• FAQs browsed to questions asked ratio: 6.4 to 1 

• Keyword searches to questions asked ratio: 6 to 1 
~ Ratios co-vary with time 

• Highlights value of "self serve" answers 

- Potential number of questions large relative to those asked 

• Nearly eliminated out-of-scope and prank questions 

• Less successful eliminating overly general questions 
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SEDAC Help Desk Implementation 

• Need to balance product support with 
reference assistance 

-Can give more/better reference assistance 
if help desk reduces time spent on product 
support 

• Will soon go live 
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Answers vs. Reference 

Are we subject matter experts giving answers, 
or are we information specialists providing 
reference ? 



-Ongoing struggle 

- Answer varies according to project 
-Practicality 

• Some questions have simple, straightforward 
answers 



-Liability 



VRD2000 



CTES1N b»u • primed pemwco CIBSIN, Cofcnba Untonty. 




griESIN 



New Issues/Future Directions 



• Evaluation, evaluation, evaluation 
-Internal motivation 

* Prompt response may not produce "archive" 
quality answer 

- Need to review before adding to public FAQs 

• Periodic review of FAQs 



- Sponsor needs 
• GPRA requirements 
- User satisfaction metrics 
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New Issues/Future Directions 

• Upcoming version of RightNow Web will 
allow us to track FAQs browsed and 
keywords searched by user before they 
asked question 

• Privacy concerns 
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A Management Model for Digital Reference 
Services In Large Institutions 

Barbara MacAdam 
Suzanne Gray 

Abstract 

Large, decentralized information organizations face a unique set of challenges in 
effective management of virtual reference service. Proposed here is a model that is 
particularly relevant to academic libraries in large research institutions, but extendable to 
other types of organizations with similar characteristics. The five critical issues examined 
include: integrating virtual reference service with existing services; allocating fixed 
resources; acting as an effective advocate to secure organizational support; developing a 
distributed service model integrating specialized, subject-domain expertise; and targeting 
and serving disparate segments of the user community. These concepts and strategies will 
be most useful to managers in information organizations who have responsibility for 
reference service development and need solutions designed specifically to achieve service 
goals for virtual reference by working within and across the larger organization. 

Introduction 

Information professionals who made the assumption that virtual reference services 
development would share most of the challenges of traditional reference models, and 
roughly the same solution strategies, have quickly realized the obvious and significant 
differences. Moreover, large, decentralized information organizations face a unique set of 
challenges in effective management of virtual reference services. A sound management 
model applied to academic libraries in large, research institutions may have useful 
application to other types of organizations with similar characteristics. The factors unique 
to the larger organization can be neutral elements, barriers, or positive agents in any 
effort to develop and manage digital information services, depending on the ability to 
manipulate them successfully in a new initiative. 

Characteristics 

1. A broad mission - A typical academic library mission statement might be to support 
the research and educational endeavor of the campus, to serve the larger community for 
the public good, and to acquire and preserve the collection for the knowledge base of the 
future. While almost any activity can be justified under this mission, it is harder for any 
particular service initiative to demonstrate mission critical importance. 

2. Various and sometimes conflicting goals - Conflicting goals might include: to 
provide timely, location-independent information assistance at the time of need; to foster 
information literate undergraduates; to serve as research partners with the teaching 
faculty; to maintain welcoming and widely accessible physical facilities; to sustain the 
culture of the book; and to foster the adoption of new information technology. Tensions 



occur when large organizations try to turn fundamental values into operational strategies. 
Digital reference service must advance some clear goal to be perceived as though it is 
worth the resources and effort allotted; these resources and services will have been 
inevitably diverted from other important initiatives. 

3. Decentralization - The work of the organization is spread over multiple units, 
separated by function (e.g., technical services, public services, systems, financial and 
personnel operations) and often isolated geographically and administratively. Large 
public service operations may have multiple physical service points, multiple reference 
departments, highly differentiated resources, variations in policy and process, and even 
unit-specific mission statements and goals. Virtual reference service, by its very nature, is 
at odds with this organizational segmentation, yet its success will depend on the 
manager’s ability to work within existing realities in a distributed service model. 

4. Centralization - The largest and most distributed organizations have centralized some 
key operations including systems support, software approval, purchasing, or operating 
budget. This can potentially hinder the manager’s autonomy in making service decisions, 
establishing priorities, and controlling resources. 

5. Widely dispersed and over-committed resources - Large organizations have large 
budgets and a large staff, but these are often deployed broadly and thinly across the 
distributed service operation. It is critical to understand the dimensions of 
decentralization versus centralization and build cooperation in order to develop 
effectively the critical mass of resources necessary to sustain a digital reference service. 

6. A non-homogeneous user community - Faculty, graduate students, undergraduate 
students, alumni, the state or regional community, a global research community, 
corporate partners, and consortial agreements may all place varying demands on a virtual 
reference service. Although their needs vary, along with the information organization’s 
perceived obligation to meet those needs, virtual reference users will inevitably represent 
a broader spectrum of the larger community than traditionally served. One of the most 
critical challenges is designing a central service gateway, balanced with a policy and 
operational structure that preserves the desired service focus for segments of the user 
community. 

7. An established set of services - Services are highly entrenched with a history of 
budget support, service structures, staffing, and demonstrated importance to the user 
community. It is unrealistic to expect existing services (and their advocates) to just move 
over or relinquish energy and resources to digital reference development. 

8. Highly-specialized subject expertise - Particularly in research libraries, the service 
structure depends heavily upon the research consultation role of subject specialists spread 
across the organization. This includes specialists of discipline-related knowledge and 
advanced technical expertise. Unless goals for the service are merely to provide basic 
information assistance, a successful model for digital reference must include ways to 
preserve and foster the links between users and the specialized knowledge providers. 
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9. A complex institutional and organizational culture - Large organizations present an 
often mystifying array of traditions, unwritten rules, territorial issues, protocols, style, 
and implicit values that hinder a clear understanding of the most effective methods for 
achieving goals. Virtual reference service grows from an environment that is dynamic, 
entrepreneurial, democratic, and inherently impatient. Sustaining the development and 
growth of a service, however, requires stability, support, and resources that ultimately 
only a broad-based, firmly anchored position in the organization can ensure. 

10. Historically entrenched user expectations - Virtual reference may be perceived by 
some as a non-essential program draining resources, time, and effort away from the work 
of more traditional reference service. Faculty may deplore efforts that permit students to 
stay out of the library physically, and administrators may worry that staff, already hard- 
pressed to meet needs of primary users, are now responding to demands from users 
around the world. 

11. Complacency, conservatism, and inertia - Large organizations are generally secure 
in their traditional success and complacent about their importance to the user community. 
The sheer volume of services and activity on a day-to-day basis can overwhelm efforts to 
identify emerging user needs and to respond in a timely way. 

12. Process - Large organizations have a process for everything, ensuring that there is a 
way to get everything done, and guaranteeing that major service innovations run the risk 
of being over scrutinized. 

13. Pride, creativity, know-how, vision, energy, dynamism, resources - All of the 
above elements reside somewhere in a large organization, especially today’s research 
library. It is critical to marshal these qualities with the same urgency with which one 
would identify and deploy more tangible resources. 

Turning Critical Issues into Achievable Goals 

Defining an Appropriate Role and Integrating Virtual Reference Service with Existing 

Services 

1. Identifying service gaps - Precisely what user needs are currently unmet? If there is a 
lack of reference service across long hours in small, divisional library units, or support is 
needed for users working primarily with digital resources, it needs to be made explicit 
why the best solution is not merely expanding existing information services. 

2. Confronting changing user patterns - Has reference desk traffic gone down? More 
than likely, it has, since students now have 24 hour access to the Web through the campus 
computer labs or residence halls, and many faculty have developed a full curriculum in 
the Web environment, sustaining relationships with and among their students in that 
environment. 
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3. Establishing a shared purpose and clear priorities - The goal is not to create a 
digital reference desk. Presumably, the goal is to provide outstanding information 
services for the user community, and digital reference service is a strategy that is 
responsive to unmet user needs. This is the foundation that will ensure strong connections 
to present services. 

4. Laying an existing service element to rest - What is the organization willing to give 
up? Put another way, somewhere in the array of information services lurk components 
that users neither need nor desire; they continue because staff is unwilling to relinquish 
them. 

5. Combining seasoned experience with creative innovation - Developing projects in 
the digital environment takes knowledge of technology, creativity, and a zest for 
experimentation. Sustaining a working service requires experience, stability, dedication, 
and sound management. Changes in the information environment and professional 
education have been so profound and happened so quickly that the potential exists in 
most larger organizations to involve veteran and newer professionals whose synergy will 
ensure successful digital service development. 

6. Applying or adapting existing policies - Existing policy typically includes 
safeguards for privacy of patron transactions, levels of privilege for different segments of 
the user community, and guidelines for referrals. Policy is designed to make the majority 
of transactions flow smoothly, consistently, and predictably from the user’s point of view 
and to ensure that users are enfranchised appropriately and treated ethically and that 
organizational resources are expended efficiently. The degree to which virtual reference 
services integrate successfully with existing services is determined somewhat by the 
degree to which users neither lose nor gain status in the digital environment and to which 
existing policies remain in new services. Inevitably, there will be areas, internal and 
external, where existing policy will be insufficient. New policies should be consistent in 
values and outcomes of existing policy. 

Allocating Fixed Resources Effectively 

1. Administrative authority and budget flexibility - Virtual reference services must be 
sustainable, and in most organizations that means redirecting resources from an existing 
budget. During an entrepreneurial or pilot phase, one-time money for capital equipment, 
temporary staffing (e.g., interns, residents, or use of lapsed salary money), or seed/grant 
money from inside or outside the institution are all reasonable ways to get an initiative 
off the ground. A viable, long-term service relies on diverting base budget resources; this 
means planning, which includes working within established processes for equipment 
requests, reassigning existing staff, or shifting money among personnel lines. The digital 
reference service manager needs authority to allocate resources or the support of an 
administrator who does, along with the experience and confidence to make aggressive 
budget decisions. 
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2. Staffing - In the larger information organization, it is not realistic to build a service 
solely on the assumption of hiring additional staff, exclusively devoted to providing 
digital reference. The best staff model integrates existing staff to the appropriate degree, 
with enough net staffing increase to make the service viable. In the search for additional 
resources, late-night reference staffing might be structured to handle virtual reference 
queries in slow periods between patrons; staff members who traditionally have not 
worked evenings or weekends might be willing to provide service from home over 
extended hours. Digital information services in the large, decentralized organization must 
invariably migrate toward a distributed staffing model. 

3. Collaboration - Fostering cooperation among individuals and units within and outside 
the library serves two purposes. It can produce a critical mass of staff, expertise, and 
other resources lacking in any individual unit. It can introduce the combined elements of 
flexibility and stability for service development. 

Being an Effective Advocate and Ensuring Administrative, Staff and Technical Support 

1 . Gaining staff buy-in - The quality and longevity of a digital reference service depends 
on the staff who provide it. It is critical that staff share a vision for information service 
and perception of unmet user needs. They should be involved in every step of service 
planning, implementation, and operation. Managers should demonstrate that they are 
conscious of the very real additional demands of digital reference. One should be clear as 
to where resources are coming from, and convey that thought is being given to long-term 
needs: from resources, to protecting existing service, to staff training. It should be made 
clear that staff will not give up their direct role with users; digital reference service can 
actually foster contact with their user community. The enthusiastic manager also needs to 
be realistic as to how quickly service development can occur and to make sure 
administrative zeal doesn’t outpace the time and energy of committed but overworked 
staff. 

2. Resourcefulness, independence and risk-taking - Ideally, every information service 
manager would find the information technology infrastructure, administrative support, 
resources, and expertise readily available to develop and sustain a digital reference 
service. Because larger organizations are usually running at full speed with staff and 
other resources fully committed, the wise manager finds ways to implement digital 
reference service using a low-key approach. The next critical phase of operations is to 
elicit higher administrative approval based on a demonstrated proof-of-concept. The 
manager who avoids requesting additional resources for every new service initiative will 
also avoid the implicit request for prior approval and the accompanying scrutiny for an 
endeavor in its early days. 

3. Determining useful allies including partners outside the organization - It is 

essential to build a cohort of colleagues who share a vision, understanding, and 
enthusiasm for digital reference. Staff at any level of the organization can be helpful in 
many ways. Overlapping goals create strong partnerships. There may be programs 
underway across campus, or within the library, where digital reference service fills an 
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identified unmet need. Community service programs, relations with alumni, distance 
education, public relations, admissions recruiting, technology support/help-desk 
initiatives or other information services, are all examples of opportunities for cooperation 
which may produce anything from resources to critical affirmation of shared institutional 
values. 

4. Clear goals and persuasive communication - In spite of the persistent belief that a 
good idea will sell itself, there is no substitute for a clear, concise statement of the goals, 
audience, and projected benefits to the user community for a virtual reference service. 
Because it is new, requires considerable development of the service underpinnings 
(software/network support, staffing, policy), and suffers from the incomprehensible but 
pervasive belief that digital content fosters completely independent users, digital 
reference service requires a justification where traditional services go unchallenged, 
despite declining statistics. 

5. Demonstrable success - Evidence of success can include steady (or exponential) 
growth in use, proof that the service is reaching a primary user community, and degree of 
user satisfaction. These will help illustrate that the service is worth the allocation of 
resources, effort, and attention required. Any successful new endeavor generates a buzz 
of good will; administrators will value and affirm a service with demonstrable significant 
outcome for the campus, colleagues will want to participate, and the user base will grow 
by word of mouth. 

6. Evaluation - The first obligation in managing a digital reference service is to provide 
outstanding service to users. Data, including patterns of use by targeted segments of the 
primary user community, have value in providing management information for ongoing 
service development and improvement. The second obligation is to provide tangible 
evidence that the virtual reference service is meeting a critical need, thus justifying the 
continued allocation of resources. The potential for research is enormous in this area, 
with a wealth of data waiting to be tapped. However, the manager who honors the first 
two obligations will ensure the success and viability of a digital reference service and be 
in a position to engage in the latter activity. 

Establishing a User-Centered Service Model that Balances Centralization and 

Decentralization 

1. One-stop shopping - Users need a central gateway to digital reference service. One 
option is to offer a button labeled “Ask Us” or “Have a Question?” on the organization’s 
Web site. This could help consolidate existing e-mail boxes and replace a variety of Web 
contact points. In a highly distributed organization, the central representation of digital 
reference services can be a significant opportunity to create a visible, easy to use, and 
consistent information service point for users. 

2. Integrating subject-domain expertise - A central digital reference service does not, 
and should not, supplant the relationship between library subject specialists and the 
faculty and students they serve. The virtual reference desk can operate as a clearinghouse 
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and referral point for users who do not know whom to contact, and actually create new 
links between librarians and users. Faculty who might normally go directly to a particular 
librarian for discipline-specific consultation may prefer to use a central service for more 
general questions or when the staff member is not available. There needs to be agreement 
across the organization that a significant amount of reference activity will continue to 
transpire directly between librarians and users, but that subject specialists will anticipate 
and be prepared to respond to inquiries through the central service. The software/network 
environment should support distributed operations. 

3. Referral structure - Referrals present a different set of problems in digital reference. 
Because many virtual reference services make a commitment to answering questions 
within a set time frame, monitoring the response time is critical. In a distributed 
environment, those specialists within the organization who are not participating directly 
in the centralized reference service may have different goals and standards for their 
service. Once the question is forwarded to another service point, the centralized service 
loses control over the quality of the answer and the response time. 

This problem can be dealt with on two levels. The first is to gain an understanding of the 
service policies for all of the specialized virtual reference service points in the 
organization. Once the policies are understood, there may be room for negotiating 
common standards of service. However, if the specialized library is unwilling to handle 
requests from those outside their primary clientele, the centralized service may need to 
avoid referring these requests to that service point. A second strategy for handling 
referrals is to clearly notify the user that the query has been turned over to the specialized 
library. Providing a Web address for the specialized service point allows the user to 
follow up on the inquiry if a timely response is not forthcoming. 

4. Setting and achieving consistent response goals - Users expect a consistent, 
predictable turn-around time for the questions they submit. Their expectations are often 
colored by the possibility of immediate access of content allowed in the Web 
environment. Typical response goals might include immediate confirmation of inquiry 
(system generated), a response or initial brief response (for more complex questions), or a 
message that the question is being referred, all within 24 hours, with a target of final 
response within one week for all questions. The best practice is to pick the shortest but 
most realistic response time and clearly communicate this to the user. Otherwise, users 
may be discouraged by a long wait or disappointed when the service fails to respond 
within the stated timeframe. Citing alternatives on the Web site (e.g., calling the 
reference desk for urgent, quick answers) will help meet user expectations. 

5. Staff training and development - Beyond reference expertise, staff participating in 
virtual reference service need to have a clear understanding of the software/information 
technology being used to support operations, and the specific goals, policies and 
guidelines for service. Within this framework, they will make judgments in constructing 
an appropriate response. As more staff and units across the larger organization participate 
in the service, it is important to build a shared understanding of service goals. Staff may 
find they prepare long responses in the virtual environment. Benchmarks are useful to 
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define how long an average response should take before a referral or other action occurs, 
and models of well-crafted responses can help even experienced reference librarians 
develop communication skills in an electronic mode. 

6. Quality control - The most fundamental way to ensure that digital reference service is 
consistently excellent is to define realistic service goals, accompanied by workable 
policies and procedures, with participating staff fully cognizant and invested in them. A 
digital reference service in a large research library must be predicated on the assumption 
that reference staff are expected to work with autonomy and to tailor responses to the 
specific situation. A well-designed Web form can elicit enough information from users to 
help offset the lack of a true reference interview. Reference styles vary among 
individuals. Staff in an undergraduate library, for example, might reasonably be expected 
to construct answers that are more instructional in nature when responding to students, 
while medical library specialists emphasize speedy, complete information to their faculty. 
Monitoring activity periodically and offering models for good responses, along with a set 
of standard answers for common requests are also effective strategies. 

Targeting Services to the Key Segments of the User Community 

1 . Needs analysis - What unmet information service needs do the various segments of the 
user community have? The first step in targeting services is to clearly define the primary 
clientele and the institutional obligations to a wider community. Undergraduates may 
need real-time, immediate assistance when working from a residence hall or lab in the 
electronic environment. Faculty on sabbatical or doctoral students at a remote research 
site, for example, have other needs. 

2. Tiered services - Because a wide range of users are attracted to a digital reference 
service, traffic from non-primary clientele needs to be controlled. Often the Web site of a 
large research library attracts non-affiliated visitors from around the world whose 
questions may not be directly related to the institution’s collection or mission. Closing the 
service off to those from the outside is one approach; however, this is often contrary to 
institutional policy regarding telephone and face-to-face reference service, which usually 
does not require proof of affiliation from users. Barriers such as time, distance, or phone 
charges minimize the number of non-affiliated users of traditional reference services, but 
virtual reference services break down these barriers by providing easy, often low cost 
access to the service. 

At most institutions, resources allocated to virtual reference services will not allow staff 
to fully meet the needs of non-primary clientele. Tiered service can create barriers for 
non-primary users while providing easy access to primary clientele. Services can be 
tiered based on response time, the amount of information requested of the user, and the 
quality of the answer provided. For affiliated users, the goal is to continue to decrease the 
response time in order to make the service as convenient as possible. For non-primary 
clientele, a more significant wait, perhaps of a week or more, may encourage them to 
satisfy their information need through another service. The use of separate forms for 
different user groups can also create barriers to non-primary clientele. Requesting 
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additional information from these users helps to provide a clearer context for answering, 
while also requiring additional effort on the user’s part to generate a query. Additionally, 
primary clientele are provided with an in depth response that is frequently instructional in 
nature, while non-primary clientele may be provided with a few Web sites or a referral to 
a specific search engine. In order for these barriers to be effective, this policy must be 
clearly communicated to users on the virtual reference Web site. Additionally, staff must 
be trained on the service goals for each category of user to maximize efficiency. 

Tiered service allows for better allocation of staff time to handle incoming queries. 
Different types of staff can be utilized to handle the different categories of requests 
generated by a tiered service. 



Current Challenges 

The Perfect Software 

A still elusive goal for large-scale information services is the complete and seamless 
integration of digital reference operations. The ideal management software would support 
Web-based asynchronous and real-time interaction and telephone and site-based, face-to- 
face reference; facilitate the exchange of digital content; provide flexibility to partition or 
centralize the environment as needed in a distributed service organization; and provide 
the tracking, archiving, search capability, and use-report capabilities critical for the 
effective management of ongoing operations. 

Intellectual Freedom, Privacy, and Access 

The digital reference environment turns each of these issues into a double-edged sword. 
Users may communicate anonymously with greater ease and may engage in more 
egregiously offensive or inappropriate behavior. At the same time it may be more 
difficult to protect an individual’s privacy when detailed information may be captured in 
the initial Web form, and queries are referred or archived. Network authentication, or 
licensing agreements, may present unexpected barriers to service access. On the other 
hand, the accessibility and visibility of the organization’s Web site will expand the user 
community, requiring hard decisions about balancing the public good with the need to 
serve primary clientele. 



Evaluation 

Virtual reference service will invariably grow even in the absence of aggressive 
promotion, but the real challenge is to demonstrate effective and high-demand service for 
the primary user community. Collecting detailed use and user statistics depends on 
adequate data capture and reporting capability within the system, but may present another 
set of confidentiality issues. The more critical need is to develop the goals, service 
performance measures, and data analysis methods for meaningful, systematic assessment 
where the range of complex activity is distributed across the organization. 
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Expanding Services 



Continual identification of unmet user needs will provoke considerations of new 
dimensions of service. For example, a real-time chat component may offer a solution for 
undergraduates who often cannot wait for virtual reference services that take one or two 
days to respond. Controlled expansion, targeted to primary user groups’ needs, will result 
in clearer strategies to reallocate resources appropriately. 

Cooperation and Competition 

Cross-institutional service collaborations, commercial information service development, 
and the development of tools and user interface designs that foster independent use of 
digital content all have enormous potential to change the virtual reference landscape. 
Large, research institutions face a unique challenge to provide substantive, highly 
specialized, discipline-related information and research assistance to a broad user 
community. Academic communities are also dependent on successful relationships 
among units and individuals working within an institutional culture. Wise management of 
digital reference services should be able to identify clearly where needs of the user 
community can be better served through cooperative initiatives or broad-purpose, 
commercial information systems. 
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Managing and Improving Digital Reference Service: Case Study 



http://vrd.org/conferencesA^RD2000/proceedings/reed-intro.shtml 
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Managing and improving Digital Reference Service: Case Study 

Donna Reed 

Multnomah County Library 

Presentation 

introduction 

Multnomah County Library launched Ask Us! Online, a digital 
reference service, to the public via its Web site 
(http://www.multcolib.org/) in September 1999, one year after its 
internal launch. At the time the service went live to the public, 
library staff had answered over 10,000 questions submitted through 
internal mechanisms. By September 2000, library staff had 
answered approximately 17,000 digital reference questions. 



The service, then called Information Dispatch, was initially 
launched in 1997 as part of a grant-funded project. At that time, it 
consisted of a Web form with no back end. It was discontinued 
after six months because the library was reorganizing and the 
central library was relocating. In 1998, the library formed a team of 
reference librarians, administrators and technical staff to begin 
investigating the possibility of further developing the project. The 
result was the creation of a Web-based database-driven system 
running on an Oracle database with a Perl scripted front end. 



The development team saw the project through to the beta stage 
and was replaced by a second team that refined the service, 
documented procedures, managed the training, and oversaw the 
internal launch. This shift happened because of staff changes and 
the addition of new staff. Recommended practice would be to keep 
one team in place and to make replacements as needed. It is critical 
that there are representatives on the team from all stakeholder 
groups throughout the system at all times. 




The first year of the service was spent further refining the service 
and exploring issues around the workflow change. During this 
time, administrative staff held a series of focus groups in order to 
hear staff concerns and make further refinements where possible. 
Once the service went live to the public, the library created an 
oversight group called PIER (Pioneers in Electronic Reference) to 
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act as stewards for the service and to make sure that concerns, 
needed improvements, policy changes and training issues were 
addressed on an ongoing basis. The PIER group also maintains an 
intranet presence and tracks statistics for the service. A database 
driven statistics program was put in place in August 2000 ( 
http://www.co.multnomah.or.us/reflib/stats/getstats.html) . It allows 
the library to track many aspects of the questions and generates 
reports on the fly. 

Currently the PIER Group and the library's reference committee are 
looking to the future to decide how the library should continue to 
handle digital reference. The PIER Group is discussing needed 
improvements and ongoing policy and procedural issues. The 
library is exploring whether to continue upgrading the existing 
system or to purchase commercial software. Staff members are 
interested in technologies supporting chat and collaborative 
browsing. The library has recently become a member of the 
Collaborative Digital Reference Service (CDRS), an initiative 
organized by the Library of Congress, in order to explore the 
possibility of a library-based reference network. (See 
http://www.loc.^ov/rr/digiref/ for details). These explorations may 
take the library in any number of directions and the only certainty 
is that digital reference is here to stay. 

| Home | Contact Us | Search Site | Site Map | For Kids | About VRD | 

| Conferences | VRD Network | AskA+ Locator | DIG REF | AskA Consortium | 
| Publications | Resources | Training [ What's New? 1 VRD Learning Center | 
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Managing and Improving Digital Reference Service 
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Donna Reed , 

Community Information Services, 
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503.988.5238 
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A brief history of the project 

• One upon a time there was a grant 

• Then we tried again 

• There were tax cuts 

• And we reorganized 

• Now we have support 
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2nd Annual Digital Reference Conference 

The Development Process 

• WebRefTeam 

o Made up of reference, tech and administrative staff 
o Developed beta 
o Tested beta internally 

• Information Dispatch Group 

o Suggested refinements 
o Renamed service 
o Created mandatory staff training 
o Created initial documentation 
o Oversaw internal launch 

• Pioneers in Electronic Reference (PIER) Group 

o Oversaw public launch 
o Manage post public launch 
o Discuss development issues 
o Maintain documentation 
o Keep eyes to the future 
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Birth of Ask Us! Online 
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• Internal launch September, 1998 

• Public launch September, 1999 

o 10,000 questions answered by public launch 

Technical info 

• Oracle database 

• Perl scripted front end 

This is what the staff screen of Ask Us! Online looks like. 
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Life after birth -- or getting past the terrible twos 

PIER Group formed -- charged with: 

• Stewardship of the service 

• Data gathering 

• Ongoing development of policies & procedures 

• Training issues 

• Envisioning the future 
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Stewardship 

• Providing a mechanism for staff input 

• Overseeing how the service is functioning 

• Troubleshooting problems 
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Datagathering 




• Worrying about number of questions asked 

o What if we're flooded with questions? 
o How do we load balance? 
o Can we respond? 

• Creation of the statistics program 

• Keeping an eye on impact to staff 
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Ongoing development of policies & procedures 

o Incorporating new policies 
o Clarifying when necessary 
o Maintaining the documentation 
■ Procedures on the intranet 
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Training 

o Development of classes 
o Updating curriculum 
o Incorporating training into orientation 
o Identifying training goals 
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The Future -- So What's Next? 

o Refinement of existing software 
o Conversion to commercial software 
o Participation in LC's CDRS project 
■ http://lcweb.loc.gov/rr/digiref/ 
o Additional forms of communication -- chat, collaborative 
browsing... 

Resources 

o A list of resources maintained by Peggy Hadid, Multnomah County 
Library 
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Tips -- or things to avoid 

o Make sure there is administrative support for the project 
o Spend a lot of time pulling together the development team 
o Spend a lot of time pulling together the development team 
o Spend a lot of time pulling together the development team 
o Spend a lot of time pulling together the development team 
o Make sure to include a lot of reference staff 
o Expect team members to communicate with staff 
o Elicit input from non-team members 
o Choose software solution late in the game 
o If building in-house, beta-test with a small sub-set of staff 
o Understand that the service will need nurturing 
o Create oversight team post-launch 
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Questions? 

Web address for this presentation: 
http://www.multcolib.org/products/vrd 
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Communication Theory and the Design of 
Live Online Reference Services 

Susan Ware 

Penn State - Delaware County 

Presentation 

Introduction 

Social Presence Theory & Media Richness Theory 

Research on the impact of emerging computer technologies on communication in 
organizations led to the development of social presence theory (Short, Williams, & 
Christie, 1976) and media richness theory (Daft & Lengel, 1986). Social presence is the 
degree to which a medium is perceived to convey the actual presence of communicating 
participants. A medium’s richness is measured by its capacity for immediate feedback, 
multiple cues, language variety, and personalization. These and more recent computer- 
mediated communication studies confirm that: 

• Social presence is essential for intense and relational computer-mediated 
communication, 

• Richer media facilitate more accurate and meaningful transmission of ideas, 

• Individuals prefer to solve collaborative, equivocal tasks through a medium that is 
able to sustain relationships and facilitate spontaneous, interactive communication. 

Live Online Reference Service 

The negotiation of complex reference queries is an interpersonal and collaborative 
task. Face-to-face reference interviews involve non-verbal cues that invite queries, open 
questions that encourage elaboration, closed questions that clarify the query, search 
strategy planning, and search demonstrations to prepare researchers to continue 
independently. With the growth of digital libraries in higher education, increasing 
numbers of researchers are searching library collections from remote locations, and the 
need for online point-of-use reference service is mounting. Online reference service that 
is comparable to face-to-face service requires real-time interactivity, high social presence, 
and media richness. The leading models of Web-based customer service software offer 
features that support high social presence and prompt synchronous interactivity. Those 
features include live text-based chat, stored scripts, Web page push, browser sharing, and 
forms/applications sharing. Easy access, authentication, and searchable archives are 
additional features that support efficient and effective management of a live online 
reference service. This presentation suggests that live online reference service is the next 
frontier for digital libraries. Only when reference librarians and researchers can engage in 
intense relational query negotiation online will digital libraries become full participants in 
technology-enhanced teaching and learning in higher education. 
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The Reference Process 
~ Research Questions ~ 


§ • 


•Invite Questions 


1 


•Probe 
•Summarize 
•Plan Strategy 
•Demonstrate 





Face-to-Face Reference 



Invite Questions 

greet, rove 

eye contact, gestures 

•Probe 

open questions encourage 
elaboration 

•Summarize 

closed questions clarify 



Face-to-Face Reference 


| *$ 3 . 


•Plan Strategy 

keywords. Boolean 
databases 
test searches 


•Demonstrate 

commands 
search results 
revised strategy 
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Computer- Mediated Reference 


/ 


A 


•Cues Filtered Out 






x Non-Verbal Cues 


1 


x Paralinguistic Cues 
x Social-Context Cues 
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Social Presence 




Medium is perceived to convey the actual 
presence of communicating 
participants (Short, 1976). 



Media Richness 







Medium's capacity for 
immediate feedback, multiple cues, 
language variety, personalization 
(Daft 4 Lengel, 1986). 
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Guiding Principles 




Social Presence <& Media Richness 


» Social presence is essentia! for intense and 
relational computer-mediated communication . 




in Online Reference 


* Richer media facilitate more accurate and 
meaningful transmission of ideas. 




Web -Based 

Customer Service Software 


+ Tasks that involve interpersonal skills and 
negotiation demand high social presence and 
media richness . 

a Individuals prefer to solve collaborative , 
equivocal tasks in a medium that can sustain 
relationships and facilitate spontaneous *, 
interactive communication. 





Web -Based 

Customer Service Software 




Easy Access/ Authentication 






c e Button Click Access 


c . Easy Access / Authentication 




hyperlink from many Web pages 


Live Chat <" • "Forms Sharing 




" e > Authentication 


Stored Scripts r Archive/Reports 
® > Web Page Push/Browser Sharing 




control access by ID login 
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Live Chat 




Stored Scripts 


C$t> 




cW) 




<■" Greetings 


<: Synchronous Interactivity 




“Hi. How can I help you?" 
“Goodbye now and good luckf 


text -based 




c •-«> Open Questions 


" Multiple Chats 




“Your topic is very broad. Can you tell me what 


queue management 




aspects interest you most?" 


e > Chat Transfer 




- Closed Questions 


among subject specialists 




“Must you use only scholarly journals or will 
magazines 4 newspapers be acceptable?" 



Web Page Push/Browser Sharing 




Forms/ Applications Sharing 


< Web Page Push 






push live URL to user s chat window 
open page in new window on user's desktop 




< Assist with Forms Completion 


Browser Sharing 

transport users to Web sites 
demonstrate database search strategies 




Share/Edit Files A Documents 




Archive/Reports 




Libraries Offering 
Live Chat Reference Services 






ELITE Project : Library Chat Services 
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■ type of library 
• chat software sites 
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' • > Searchable Transaction Archive 




UveRef(sm): A Registry of Real-Time Oigital Reference Services 


•share transcripts with users 




■ type of library 
• chat software sites 


•use for reference training 

•track problems 

•prepare management reports 








Live Online Reference Service 


Social Presence 


•Invite Questions 
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•Probe 




•Summarize 


Media Richness 


•Plan Strategy 
•Demonstrate 
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Current Research in Digital Reference 



Joe Janes 

University of Washington, Seattle 

Presentation 

Introduction 

This presentation provides a summary of research findings and ongoing projects 
in digital reference. Included are studies on the nature and scope of digital reference 
services in public and academic libraries, surveys of attitudes of reference librarians 
about technology and reference, and notions of how reference practice may be changing. 
Questions for future research are explored as well. 
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features and aspects: 


Current Research in Digital 
Reference 




the Web 

“The Web as a Reference Tool: Comparisons with Traditional 
Sources” (\v/ Charles R. McClure), Public Ukraiies J#{1 ), 30-39, 

| an u ary /Pc br unry 1 999 






accuracy and speed of answers roughly comparable 
Web: more sources used (3.56 v. 2.77) 


Joseph Janes 

Coordinator , MLIS Program 
The Information School 
of the University of Washington 




attitudes: non-Web sources judged slightly more 
authoritative, of higher quality 



features and aspects: 
academic libraries (AL) 

"Digital Reference Services in Academic libraries” (w/David Carter 
and Patricia Memmott), in Reference and User Sendees Quarterly }9 (2), 
145-150, Winter 1999 

45% of academic libraries offering digital ref service 
larger libraries more likely to 
Vz linked from front page, mostly e-mail/simple web form 
policies: turnaround time, users, questions (each >.50) 
public schools more likely to have a service, policy on 
questions; private schools more likely to have tech 
barrier 



features and aspects: 
public libraries (PL) 

replication of academic library study in PLs 

n — 352, stratified by population served, >10,000, > 1 librarian 

Web sites investigated March /April 2000 

81% of PLs had Web sites (293) 

of those, 64 had a service (12.8% weighted overall) 

56% directly linked from home page (44% not) 

E-mail/simple form most common, detailed form 25%; technology 
more sophisticated as size of community increases 
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features and aspects: 
public libraries 

lower incidence of policies 

highest incidence in largest libraries (39%); lower in smaller 

( 10 %) 

very few FAQ/FARQ pages (9), mostly policy 
detailed form questions: where live, phone number, 
grade/age/level, need-by date, sources tried 
also: where did you see this, company/institution, 1 st time 
user?, library card #, branch 



features and aspects: 
public libraries 

other things 

if you need quicker help, call; genealogy is special (call, come in, 
regular mail only, go to historical society); confidentiality; how 
to get an e-mail account 

2 forms exactly the same 
policy on users: 

community residents only, or questions about 
community/area/collections 



features and aspects: 
public libraries 

names: diversity, jargon, inconsistency 
28 different titles at top of pages (most frequent Ask a/the 
Librarian) 

2 5 required 2 clicks to get to page, 1 8 different names on 
home pages (incl. “Feedback", “Adult Services", “Using 
the Library") 

17: name in link is different than title of page sent to 
12: 3 different names 



experiences, 
opinions and attitudes 

national survey of reference librarians 
n= 1548 (cluster sample) 

5-page survey 

648 responses rec'd (RR = 42%) 
preliminary results ONLY 
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experiences, 
opinions and attitudes 

3 /4+ have used e-mail for reference, Vz Web forms, very few 
other technologies (chat, MOO, video) 
most likely to agree that digital technologies make 
reference: 

more accessible, more interesting, more challenging, more fun 
least likely to 7 agree that digital technologies make 
reference: 

cheaper, more difficult, more time consuming 

very similar pattern of responses with “use of digital resources” 



experiences, 
opinions and attitudes 

digital reference will best serve: 
ready reference Qs 
Qs from regular library users 
Qs in popular culture 

digital reference will most poorly sew. 
research Qs 
Qs from children 
Qs of a personal/private nature 



experiences, 
opinions and attitudes 

# of reference questions received is slightly decreasing 
(1/3 decreasing, 14 staying same, 14 increasing) 
questions are getting harder (1/3 harder, t/10 easier, Vz 
staying about the same) 

Internet training: in current position (4/5), in degree 
program (3/10), in previous position ( 1/4) 
attitudes change with experience 



the I PL reference service 

begun March 1995, part of Internet Public Library project 
answered 20,000+ questions 
any and all types of questions, users 
examined questions received by IPL January - March 1999 
looking for ways in which questions are submitted, types of 
questions, processes in answering, thanks, etc 
automatic processing of files, no in-depth examination yet 
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IPL results 

3,022 questions received; 68% via form (“interview"), 26% 
via e-mail, 5% other forms 
most frequent subject for question chosen by user: 
Other/Misc (42% of questions from form) 
next: Education, Science, Humanities, Government/Law, 
Business, Libraries/Librarians (3-6% each) 

24% identify as businesspeople, 1 1% as teachers, 7% as 
librarians, 36% as "for a school assignment" 



IPL results 

users on form asked to specify whether they want factual 
answer or sources to help in answering 
IPL administrators have the option of reversing this decision 
based on their expertise, opinion 
reversed 7% of "sources" to "factual" 
reversed 40% of "factual" to "sources" 



IPL results 

the software used by IPL allows question answerers (students, staff, 
professional volunteers) to post internal-only messages 
(followups) 

34% of questions had 1 or more followups from the answerer (max 
10, avg. 0.63) 

25% of questions had 1 or more followups from others (max 8, avg. 
0.44) 

other internal flags: NEED JELP (3.2%), ASKJNFO (4.0%) 

30% of patrons asked for more information never responded 



IPL results 

time to answer (from question received to answer sent) 
average 2.96 days (s.d. 2.70) 

Q1 (25%ile) 0.77 

median 2.05 

Q3 (75%ile) 4.89 

factual: 2.10 days, sources: 2.31 days (no sig diff) 
fastest: factual questions from e-mail (1.69 days) 
slowest: sources questions from e-mail (2.38 days) 
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IPL results 

unsolicited thanks received on 197% of questions 
factual 24.4%, sources 18.0% (sig at .001 ) 

Thank rate by subject area of question: 

LIB 26.2, MUS 25.9, EDU 23.8, LIT 22.3, HUM 22.2 
PF 4.0, FARQ 3.0 



IPL results 

Thank rate for questions answered in <2.05 days (median): 1 8.4% 

Thank rate for questions answered in >=2.05 days: 22.6% (sig .01) 

Thank rate for questions answered before posting: 1 0.8% 

Thank rate for questions (not answered before posting) with 1+ followups: 
28.9% 

Thank rate for questions with no followups: 1 6.9% 



analysis 

users have difficulty with assigning subject areas, deciding 
on nature of answer 

e-mail possibly good for quick questions, form/interview 
better for sources/research type questions 
dropout rate: due to getting answer elsewhere, thin 
connection with users, they don’t care?? 
thanks highest from factual questions, humanities areas, 
librarians and businesspeople, questions that took longer 
to answer, with more internal activity 



evaluation of ask-an-expert 

services 

report to come in following session (Hill & Rolfe) 
sent questions to 20 .com and 20 .org sites 
factual, sources, scope Qs 

look at response rate, time to answer, answering Q asked, 
verifiability, would use again, characteristics of services 

fascinating results 
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thoughts & questions 

45% of ALs, 1 3% of PLs doing digital reference — should 
that be higher? 

Vz not linked (hiding) — why? should they be? 
bigger libraries have more services, more tech — resources 
are more important but not overwhelmingly 
no FAQs — why not? 

minimal interviews (25% of PLs detailed forms) — why not? 
stop weaseling (time policies), confusing (name changes) 



themes 

reaction of more stuff and greater use of stuff 
reflection of setting, clientele, expectations, context 
facilitation, empowerment, education of users 
adoption of technology 

librarians are ready (training, interesting/cha!lenging/fun) 
but don’t see panacea (no cheaper or quicker) 

limitations, boundaries, policies sticking in our toes 

hiding, confusing, weaseling (yet accessibility 1 st on survey) 
fewer harder questions 



implications 

maybe fewer harder questions is the answer 
easier to ask questions, different kinds of questions (harder, 
"research" questions; ok (?) to have slower response times) 
use technology as medium and tool 
optima] allocation of most precious resource 
rethink the "reference transaction" as an ongoing process 
partnerships with experts 

break the boundaries of library as place yet maintain the values, 
heritage, knowledge there 
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Digital Reference Standards 



Brett Butler 
Infour 

R. David Lankes 
Information Institute of Syracuse 

Introduction 

With the growth of digital reference services, more and more Internet-based 
questions and answers are generated in a variety of formats. This section focuses on the 
need for standard representation of digital reference transactions for tracking and sharing 
this valuable data. Two current projects to define standard formats for digital reference 
data are highlighted: the Knowledge Bit, from AnswerBase and other organizations, and 
Question Interchange Profile (QuIP), from the Virtual Reference Desk Project. 
Implications of these standards development activities are also discussed. 

KnowledgeBit: A Database Format for Reference by Brett Butler 

Question Interchange Profile (QuIP): Metadata for Cooperative Reference by R. David 
Lankes 



KnowledgeBit: A Database Format for Reference 

Version 2.0 

Brett Butler, Publisher 
AnswerBase.net 

Abstract 

This paper examines the need for a common, standard data format for the management of 
reference transactions. This need is discussed from two perspectives: that of major 
research libraries and that of AnswerBase Corporation (ABC), a Web-based digital 
publisher. 

In library terminology, reference transactions are the questions and answers that flow 
between patrons and staff; these transactions increase through electronic means. To 
AnswerBase Corporation, a new publisher that captures and organizes questions 
commonly asked of librarians, these transactions cumulatively represent the patrons’ 
demand for information in a new and powerful way. This paper describes and defines the 
Knowledge Bit reference format, designed jointly by AnswerBase libraries including the 
Library of Congress and the National Agricultural Library, to capture quality content 
from their reference work. 

A Combined Format for Access and Delivery 
Queries, Responses, and Metadata 

In order to develop its AnswerBase reference desk support and publishing system (see 
www.answerbase.net) . ABC undertook a review of the literature on the reference process, 
the reference interview, and the related value added processes. Based on results from the 
literature review, ABC designed a database format (with notable help from the Library of 
Congress’ Collaborative Digital Reference Service, which organized a design and 
planning seminar in September 1999) that was capable of growth yet produced accurate 
answers. This effort resulted in the creation of the Knowledge Bit (KBIT) format, which 
defines a new kind of standard. 



Queries and Questions 

A directly framed question (e.g., “How deep is a fathom?”) has a direct answer (e.g., “six 
feet”). An equally direct question (e.g., “How deep is the ocean?”) does not have a single 
direct answer. The appropriate response would be an essay on the depth variations in 
oceans, a qualified response discussing individual oceans, or a reference to a more 
detailed or authoritative source discussing the whole issue. Conversely, there are implicit 
questions (e.g., “My car won’t start”), which an automated help-desk system would 
translate into a question (e.g., "How do I get my car started?”). This paper describes both 
questions and problem statements as queries within the KBIT format. 



Responses and Answers 



Answers may be defined herein as effective responses to a problem statement. Answers 
that are objectively agreed to be accurate or factual fit within this definition (e.g., a 
fathom is six feet). Answers that represent expert or informed knowledge (i.e., answers 
by someone holding appropriate knowledge) are answers even if they are long, 
conditional, or express limitations. Answers that only guide or direct are at the outer 
boundary of the definition. Opinions, chat, and gossip are not considered answers. 

Metadata - About the Q&A 

One of the shortcomings of traditional card or even local computer files of reference 
knowledge is the lack of metadata, information about the knowledge that can be used to 
retrieve it. In the absence of well-defined, rich metadata, such knowledge collections 
cannot scale to large size. Without a large scale of content, the value of any such 
individual collection is limited. 

We therefore address an innovative variety of metadata in the Knowledge Bit format. If 
we are to capture any information that may flow across a virtual or physical library 
reference desk and are to add our knowledge and organize it for future use, we must 
create an exceptionally wide, high, and deep collection of rich content. 

Questions and Answers: Capture, Identification, and Taxonomy 

Questions 

The first step in managing questions is to recognize the classic reference complaint that 
“the patron never asks the right question.” KBIT captures significant content about the 
question once it has been clarified. 

The KBIT format recognizes three types of questions: 

• The original (patron) question or query 

• Reference-interview process questions, aimed at eliciting the question 

• The formal question, or the question to be answered 

We want to capture the original question, because it may provide words or phrases that 
can be used later to link patrons’ language or terminology to that used in our retrieval 
schemes. 

KBIT also provides for identification of the question by two other classes: 

• The purpose of the question for the specific patron 

• The type of question being asked 
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This is done partly to provide context in the initial research to answer the question, and 
also to provide further guidance and understanding about the resulting question-answer 
combination. 

Finally, a set of descriptive fields allow for identification of question characteristics in 
order to manage question content within the resultant database: 

• Geographic origin 

• Language of question 

• Copyright status 

• Availability status 

In addition, depending on the application, it is possible to include all the “knowledge” 
information - classification, evaluation, and statistical analyses - to questions as well as 
to the answers with which they will normally be associated. These include: 

1. Topical classifications 

2. Key terms, classified but uncontrolled 

3. Facets or aspects 

4. Item, author, and source evaluations 

5. Frequency and relationship statistics 

The use of these tools is discussed below primarily with regard to answer content. 

Answers 

Answers, the core content of any KBIT-based database or service, have a potentially 
more complex structure. Of course, the answer record captures the content selected as the 
response to the reference question. There is also the taxonomy of the answer to be 
identified. The form of the answer is classified for later use in selection: 

• Brief answer — sufficient but to the point 

• Summary statement — perhaps from a longer answer 

• Expanded answer — probably based on those above 

• Comprehensive answer — extended answer 

• Multimedia answer or partial answer related to the above 

These forms illustrate an important principle: a single question may have many answers 
that differ in length and other characteristics (e.g., intended audience). Answers can also 
differ in the type of response provided, depending on their purpose and the information 
available: 

• Fact — the most self-contained and complete response 

• Direct answer — attempts to answer the question within itself 

• Citation — points to an answer (e.g., “look in...”) 
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• Guide — any form of pathfinder or tutorial; this response may teach the process of 
finding an answer as well. 

• Referral — may lead to an answer, but does so only by pointing to a possible 
source. 

By organizing answers into these categories, we learn what type of content the reference 
librarian intended to provide, and we can provide the appropriate type of answer to future 
requestors. The source of the answer is captured, either in standard bibliographic format 
or with URL or other Web identification methods. Other parameters of language, source, 
and intellectual property status and availability are also included, as with the question 
records. 

Topical access options and the evaluative data are of great importance for location of 
specific answers in large compilations of answer data, such as AnswerBase. It is 
important to recognize the context in which an answer is provided. Therefore the 
“knowledge bit” needs to carry better search and other metadata, since the item itself is 
relatively spare (but precise) in information content. 

Multi-Classification As a Strategy: How Do You Shelve a Digital Item? 

Classification provides the highest level of organization in an electronic environment, for 
unlike subject access, a class scheme provides orderly navigation among broader, 
narrower, and related topics. The KBIT format provides for assignment of multiple 
classification terms from multiple classification schemes for a single answer record. This 
approach is not feasible for shelf-based library systems where classification is used as the 
basis of shelving. Although MARC does provide for multiple assignments, they are rarely 
created because the book must be shelved in one place only. The Anglo-American focus 
on classification as a process entirely linked to shelf arrangement has similarly prevented 
the use of multiple class schemes as topical access tools. 

KBIT answers will not be placed on a library bookshelf, so we are free of these 
restrictions. The KBIT format allows for: 

1 . Selection of a desired classification scheme 

2. Input of a selected term from that scheme 

3. Repetition of this process for a given item (answer) 

This provides considerable flexibility in application. Individual applications will provide 
different lists of class schemes or thesauri supported, different degrees of automation in 
providing reference to and input of these structured databases, and different levels of 
support for individual groups to create and support local schemes. 

In addition to use of primary terms provided in classification schemes, the KBIT format 
allows for extended input of controlled and uncontrolled cross-references tied to the 
classification scheme(s) employed. For each classification term selected for an item, the 
following choices for related terms may be provided on an optional basis: 
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Class/Thesaurus Links 


Controlled Links 


Uncontrolled Links 


Broader Term 


Within Scheme 


Local Option 


Narrower Term 


Within Scheme 


Local Option 


Related Term 


Within Scheme 


Local Option 


Don’t Use 


Local Option 


Local Option 


Use Instead 


Local Option 


Local Option 



This capability allows for the management of local choice, for the input of local terms, 
and for user-level modification of specific KBIT content, while retaining the critical 
intellectual structure and coherence of the classification schemes. This approach also 
allows for the use of Web-based hyperlinks as a specific case of navigation links. 

The flexibility of the structure will allow interoperability of any specific or local scheme 
so long as it is related to a recognized classification or thesaurus scheme. 

Quality Evaluation As a Principle: Preserving Reference Expertise 

Bibliographic records typically have not included evaluative or quality ranking content; 
MARC treats all books equally. However, libraries and reference librarians make quality 
and value judgments every day in collection development and recommendations. The 
KBIT format enables this process to be captured and quantified for use in making 
automated recommendations to patrons directly from the database. It is particularly 
important to capture these evaluations because KBIT answers will often be provided 
without the traditional quality context to which readers are accustomed: the publisher’s 
logo, the book jacket biography of the author, the presence on the library’s shelves. 

The KBIT format includes four levels of quality evaluation for sources of information 
provided as answers: the item itself, the work from which the answer was taken, the 
author of the work, and the publisher of the work. For instance, the content of the item 
can be ranked on a scale of 1 to 10 (1 is highest) relative to: 

• Quality — overall quality of the content provided 

• Assurance — confidence in the answer’s content 

• Accuracy — precision and detailed accuracy of the answer 

• Audience — intended audience for the content 

• Width — focus of the content provided 

• Breadth — comprehensiveness of the content 

• Height — complexity or depth of the content 

While highly arbitrary and subjective, KBIT format users will be able to develop 
quantitative ranking and rating schemes for content based on compilation of evaluations 
from the original library reference selectors, subsequent publication editors, and readers. 
A similar evaluation of works can be provided, depending on specific applications. 



For authors, a separate KBIT evaluation is available, which will allow quality ratings to 
be applied across the scope of an author’s works. The evaluation categories are: 

• Qualifications — accreditation, certifications for topics discussed 

• Clarity — rating of presentation, style, language 

• Bias — existence of any visible unstated perspective 

Evaluations can be applied to individual answers or works, and cumulated on the 
applications level to provide an overall assessment of individual authors. 

Finally, well-established library criteria and evaluations can also be captured for 
publishers in KBIT to allow support in the review of unknown works from known 
sources. Quality criteria categories here include: 

• Quality — overall quality of the content provided 

• Assurance — confidence in the answer’s content 

• Accuracy — precision and detailed accuracy of the answer 

• Audience — intended audience for the content 

• Value — relationship between cost and content 

Facets for Flexibility: Content Analysis for Digital Retrieval 

In addition to the use of the structured or controlled classification and thesaurus schemes, 
the KBIT format design recognizes the uniquely broad nature of its corpus by providing 
two additional search-oriented fields: a KeyAccess classification of otherwise 
uncontrolled vocabulary, and a faceted classification scheme structure. 

KeyAccess fields could be used in various applications in a wide variety of ways. The 
intent is to provide a quick and easy input of key terms that will aid in identifying an 
answer to a future question. The challenge is to do this without restraining the input to 
authorized lists or authority files at the time of input. (This differs from applications that 
subject this content to authority processing, thereby increasing quality and consistency.) 

KBIT currently provides for KeyAccess fields in six categories: 

1 . Company — names of corporations, divisions, etc. (as used in query or answer) 

2. Person — personal names, nicknames, etc. (used in absence of name authority) 

3. Place — location names (not captured in a geographic facet) 

4. Product — brand and common names for products (could select desired product 
from a list) 

5. Phrase — linked multiword terms describing answer or object (for instance, can 
serve as a “title” for an image) 

6. Words — key search terms not otherwise classified above 
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The intent of providing these tagged fields is to “highlight” content for retrieval that 
otherwise could be lost in a full text search of answer content. These fields also serve as 
“candidate” terms for applications that operate authority control facilities. 

The KBIT format also provides for the classic six facets or elements of a bit of content: 

• Who — any classified list of names, authority or “as-used” 

• Where — any geographic classification . 

• When — any temporal classification (centuries, decades, weeks, etc.) 

• Why — classification for “explanation” answers 

• What — answers that provide description, definitions 

• Which — answers that provide recommendations 

• How — answers that explain functionality 

Tagging content with these facets will allow retrieval of answers limited to those 
components or aspects. Multiple assignments may be made, or application programs may 
create separate records where multiple aspects exist. 

Advanced Retrieval Opportunities: Beyond OPACs and Web Search Engines 

In developing the KnowBit knowledge format, AnswerBase has drawn on past 
experiences in bibliographic and information retrieval, with the understanding of new 
techniques, processes, and economic means of processing. This has led to thoughts of 
new retrieval processes that can deliver better answers to individual patrons. The 
following terms represent some areas of interest at AnswerBase as illustrations of the 
potential power of the KnowBit knowledge format. 

Classification Browsing — Librarians know that massive amounts of knowledge are 
buried in the traditional classification schemes and thesauri, but traditional systems have 
done little in this area. KnowBit stresses classification as the appropriate form of 
structured knowledge for large amounts of content to be used in a Web-based 
environment. The most advanced research in this area is being performed by Dr. Michael 
Buckland at the University of California (http://www.sims.berkeley.edu/research/oasis/') . 

Dynamic Relationships — The KBIT focus is different than that of the traditional 
bibliographic record and the work it describes. A question may have many answers, 
depending on the focus and purpose of the response. One answer may be the right 
response to many questions. While the KBIT format allows for identification of the initial 
relationship between a question and its answer, in practice applications will be able to 
recognize many statistical and behavioral patterns, thereby guiding searches. 

Feedback Systems — While the use of information feedback is used in some Web 
retrieval systems (e.g., “find more like this”), it is not a common approach in professional 
information applications such as library catalogs. Development of proactive feedback 
services is particularly appropriate to the discrete “knowledge bits” of KBIT. 
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Personalization — Made popular in the Web world by Amazon.com, personalized service 
is more a concept than a reality in most information retrieval applications. Development 
in the library environment may be slowed by libraries’ traditional protection of personal 
privacy, but Web-based services will not hesitate if it results in gaining a competitive 
edge. 

Search Guidance — Since answers are one form of guidance, and guides (such as 
pathfinders or bibliographies) are one form of answer, any KBIT application is well 
positioned to provide users with proactive search guidance well beyond that found in 
standard indices or catalogs. AnswerBase plans to include search guides internally and to 
provide links as guides to external “answers” including information vendors and expert 
research agencies. 

Since few operating retrieval systems exist that can provide these processes to a KBIT 
database structure, the argument for development of a standard is strong. 

Why not MARC or XML? The Need for a New Knowledge Format 

The scope of library reference in terms of a digital database is somewhat staggering. The 
reference librarian may need to identify the most specific fact, a mere data point, or 
content from any work in any library. At the same time, the entire scope of the Web is a 
potential, if unorganized, hunting ground. The mere combination of these two kinds of 
knowledge resources in a single database requires an approach not satisfied by use of any 
existing knowledge scheme. 

MARC was, after all, created to identify books, although the design anticipated other 
media and publishing forms. It can be extended to include periodical indexing with 
difficulty, although it has been applied to reports, maps, and more published resources 
than any other such standard. XML is being developed to bring some order to Web sites; 
it is ignorant of any data, digital or analog, not in Web form. The efforts to insert Web 
links inside MARC (the 856 field) and to provide XML forms for MARC content only 
highlight the media-dependent nature of their origins. 

More importantly, while MARC provides a descriptor to a work (historically, a physical 
object), XML and Web-based tools must wrestle with location and delivery of a highly 
ephemeral digital resource. In print and online, we face the desire for information in 
smaller and more discrete packages than the traditional: chapters or paragraphs rather 
than books, and journal articles or extracts rather than issues. 

Web services have reluctantly turned to the intellectual organization of content in view of 
the demonstrated failure of raw computer-based “searching.” It thus seems important to 
use the great organization schemes of libraries and professional societies: not only the 
Library of Congress’ Classification and Subject Headings schemes, but also those of 
MEDLINE and others. And for the great library-based classification schemes, the digital 
collection offers the heady option of multiple classifications for a single item, creating 
numerous virtual shelving schemes. 
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Recognizing these general trends, the effort to provide a truly comprehensive scheme for 
capturing bits of information tied to reference transactions became compelling to 
AnswerBase and to a number of research libraries in 1999. The result is the Knowledge 
Bit format discussed in this paper. The nomenclature is from two guiding elements of the 
design: it should be capable of dealing with discrete bits of information as provided in a 
reference transaction, and it should embody librarians’ knowledge, not just the 
information, in the form of selection and evaluation of the source information. 

Incidentally, AnswerBase’s KnowBit content will be mapped into both MARC and XML 
for use in systems employing those schemes. 

Toward a Reference Standard: A Joint Publisher and Library Initiative 

AnswerBase Corporation, as a new Web-based publisher, is acutely aware of the amount 
of content that users accept as answers to questions. As librarians and information 
professionals, we believe quality content at the “answer” level will not become widely 
available until it is possible to describe, evaluate, and characterize content in ways such 
as are made possible by the KnowBit structure. 

Therefore we plan to submit the Knowledge Bit format, as defined by the company, the 
libraries involved in its definition, and our development partners, to the National 
Information Standards Organization for review as a standard for capture of reference 
content. We are assembling a group of library reference, management, and technical 
committee members to work with us and other publishers and rightsholders. (For more 
information, contact the author at www.answerbase.net .) 

The QuIP Standard 

Question Interchange Profile (QuIP), developed by R. David Lankes of the Virtual 
Reference Desk Project, is a threaded data format that relies on metadata to maintain, 
track, and store questions and answers in a consistent file format 
http://www.vrd. 0 rg/Tech/OuIP/l .01/1 .01 d.htm) . AnswerBase believes that QuIP could 
blend well with KBIT. 



Note: Since the presentation of this paper, a formal submission for development of a 
Digital Rights Management (DRM) standard has been forwarded to the National 
Information Systems Organization. 
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Ask-An-Expert Services: Analysis Proposal 

Principle Investigator: Joseph Janes 
Research Assistants: Chrystie Hill & Alex Rolfe 

Presentation 

Introduction 

This study seeks to analyze ask-an-expert services on the Internet. These services accept 
questions asked of experts in specific disciplines, professions or specialties and can be 
offered or developed by organizations other than public, academic or special libraries. 
Particularly, these services will be assessed for accuracy, turnaround time, service 
orientation, their own evaluations, the nature of their responses and their response to out- 
of-scope questions. This study will also gather information on the services’ stated policies 
on professional advice, homework questions, limitations of service, evaluations and 
frequently asked questions. 

The purpose of this study is to determine the nature and quality of responses to user 
questions from Internet services developed and staffed by subject experts. With further 
research, assessments of ask-an-expert services can be compared with digital reference 
services currently offered by librarians on the Internet. 

Methods 

Users: Researchers will ask questions based on pre-determined criteria in each of the 
subject categories from pseudonymous hotmail accounts. Three questions will be asked 
of each of the non-commercial expert site, and each of these questions will be asked of 
the commercial sites. 

Service Sample: Ask-an-expert services fall generally into two categories. The first 
contains those services that are provided by altruistic organizations or individuals who are 
interested in digital reference (but are not necessarily librarians). These experts are 
volunteers from a huge range of disciplines and professions. The characteristics of these 
services vary. [The Virtual Reference Desk’s AskA+ Locator 
(' http://vrd.org/locatoi7subiect.shtml) contains a collection of over 80 ask-an-expert 
services that answer questions from the K-12 education community and others.] Also 
present within this category are individual sites where a person (sometimes with no 
professional affiliation whatever) deems him/herself an expert, creates a Web site and 
answers questions from the public. The other category contains commercial sites run by 
for-profit organizations. Their “about us” pages introduce investors, board members and 
CEOs. These Web sites frequently contain advertising. Samples will be drawn from each 
of these two types of ask-an-expert sites. Ten non-commercial sites and ten commercial 
sites will be evaluated. 
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Questions: Questions fall into three categories: fact, source, and out-of-scope. For the 
most part, they will be gleaned from the IPL archives in order to maintain authenticity in 
the research. These are real reference questions that have been previously asked by 
patrons in a digital environment. 

Evaluation: Evaluation will include information on the sites and services and their 
individual answers to the questions we submitted. 

Evaluation of the sites will primarily characterize the site’s functionality. Researchers 
plan to answer the following questions: 

• How much time is taken to submit questions? 

• How are questions submitted? 

• What kinds of things are required of users in order to submit questions (e.g., e- 
mail address, login name, etc.)? 

• Must subject areas be identified to answer? 

• Are there FAQs? 

• What are their policies, if any are stated? 

• 

Evaluation of the answers will include the following: 

• How much time is taken for a user to receive an answer? 

• Was further information requested? 

• How long was the answer? 

• What kind of information did the answer contain (e.g., sources, referrals, factual 
answers, etc.)? 

• Did the expert answer the question asked? 

• Is the answer verifiable? 

Prepared by Chrystie Hill 
Last updated 15 June 2000 
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Our Perspective 

• Students 

- Research conducted under the direction of Joseph Janes, with 
support from the Library of Congress and University of 
Washington (UW), 

• Reference Librarians 

- Interested in the reference exchange in variety of 
environments. 

• Researchers 

- Interested in the range and scope of information services, their 
characteristics and value. 



Current Reference Practice 



• The WWW has transformed reference from “in person’’ 
correspondence to the exchange of information in 
digital environments. 

• While digital reference is practiced by most academic 
and some public libraries, many digital reference 
services are initiated by commercial or educational 
organizations. 

• These kinds of services are referred to as “expert 
sites,” “knowledge networks,” “information exchanges” 
or "AskA services." 



What Is an Expert Service? 



• A commercial or organizational service that 
offers free or fee-based “expert" answers to 
your questions in one or several subject areas. 

• Depends on the following notions: 

- the Web is not easy to search 

- people want and should be able to ask questions of 
real people in the digital environment 








And. ..Business is Booming 



• Upside Today recently reported 
“There's no replacement for the old- 
fashioned face-to-face ... at least, not 
until now." 

• ASkme.com boasts 1 in unique visitors 
per month since April of this year. 

• Exp.com has over 100,000 experts on 
site, and invites clients to ask 
questions from “whatever subject you 
choose." 

• Datamonitor estimates 1 0m increase in 
unique users and 4m increase in 
unique experts, with revenues over 
S6b by the year 2005. 
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What’s at 


Stake... 




• It has been said that these kinds of sen/ices 
are changing permanently 

- where reference takes place 

- the way it is practiced 

- standards of information sen/ice 

• If that’s so... the very thing that librarians do 
(traditionally) is swiftly moving away from our 
desks and onto a digital stage. 

• This leads to questions about these sites and 
the kinds of sen/ices they offer. 



Preparations 

Pilot: Fall 1999 

- Researchers located as many expert sites as 
possible on the Web. 

- Reference student volunteers were asked to submit 
total of 150 questions to 20 sites. 

- Survey asked students to characterize their 
experience with the expert site and evaluate the 
response they received. 



What we learned... 



• Generally, sites and their responses highly varied in 
terms of service and quality. 

• Implications for methodology: 

- Number of sites and questions must be limited. 

_ Questions needed to be "real" digital reference questions. 

- Identifying the nature of the sites should be independent of an 
evaluation of their responses. 

- Describing experience with the sites is subjective and difficult 
to interpret. 
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I 

First of all 


, we... 




• Established criteria for 

- determining subject areas 

- developing questions and verifiable answers 

- selecting sites 

- creating characterization and evaluation measures 





Purpose o 


if this Research 




What is going on out there? 

- Characterize digital reference services on the Web. 

- Characterize sites for levels of responsiveness. 

- Analyze the quality of digital responses to these 
questions. 
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Selecting Subject Areas 



Representative of the types of reference questions likely to be asked 
in a digital environment. This was determined by the availability of 
expert sites in a subject area and common topics of known digital 
reference inquiries. 



• Classics 

• Religion 

• Literature 

• Science 

• Art 



• Math 

• Health 

• Law 

• Dinosaurs 

• Education 
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Selecting Questions 




One fact question and one source question were developed for each 
subject area. One out-of* *scope question was developed for all sites. 


• Fact Criteria 

• single questions 

• verifiable answers 

• central to the .org service subject/mission 

• asking for a specific fact or answer 

• could be "unanswerable" 
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Selecting Questions 



• Source Criteria 

• central to a .org service subject/mission 

* asks for help, guidance, sources (explicitly), not a 
specific fact or answer 

* background may be included 

• Out-of-Scope 

• appropriate for any kind of subject/site 



Sample Questions 



• Shakespeare/Fact 

- “Which play of Shakespeare’s has the moving call to battle that 
refers to ‘heroes of Crispin day’? Thanks for your help.” 

• Health/Source 

- “Can you help me find information about various doctors? I 
would like to know where they received their degrees, and any 
other information. I live in Florida, but also want information 
about Alabama doctors as well. Thank you." 

• Out-of-Scope 

- “What is the meaning of life?" 





Selecting Sites 




Selection of an equal number of both commercial and non-commercial 
sites for the purposes of evaluating these services separaleiy, and 
possibly comparing their results. 


• Commercial Sites 

- Multiple page sites with multitude of experts answering 
questions from self-identified categories 

- Diverse subject areas; can even declare your own 

# Non-commercial Sites 

- Smaller sites, limited personnel field and answer 
questions 

- Subject specific 



Naming Names 



• 10 Commercial Sites 

- About 

- Abuzz 

- AllExperts 

- Answer Poinl @ Ask.com 

- AskanExport 

- Askme 

- Exp 

- ExpertCentral 

- Frenzi 

- Knowpost 



• Non-commercial Sites 

- The Oracle 

- St. Nick 

- Shakespeare Homework Help 

- Dr. Universe 

- Joan of Art 

- Go Ask Alice 

- C-span 

- Dr, Math 

- Dino Russ 

- AskERIC 
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Characterization Criteria 



• All evaluations were submitted through Web 
forms and stored in UW database until all were 
complete. 

• Three Phases of Data Collection 

- Survey 1: Collect Site Information 

- Survey 2: When Question Submitted 

- Survey 3: Once Response Received 



Survey One 



INFORMATION ABOUT THE SITES 

• Do you have to identify a subject area in order to get to 
your expert? 

• How is a question submitted to the expert or expert 
service? 

• How much time does it take, in minutes, to enter the 
site, locate an expert or subject area, and submit a 
question? 

• What is required to submit a question to this expert or 
expert service? 



Survey One 

INFORMATION ABOUT THE SITES 

• Are FAQ pages present? 

• Does this expert service or site describe policies on 
any of the following characteristics of their service? 

• Does the expert site or service request any of the 
following information about you or your question? 

• Are questions immediately acknowledged (by e-mail or 
Web notification) that they have been received? 



r 

Survey Two 

WHEN QUESTION SUBMITTED 

• What was your exact question as stated to the expert 
or expert service? 

• If you had to identify a subject area, what subject did 
you choose? 

• What was the date and time that the question was 
submitted? 



o 
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Survey Three 



RESPONSE RECEIVED 

• Date/time response was posted. 

• Did the service or expert request further information? 

• Exact response of the expert or expert site. 

• Word count 

• Is the response to this question confined to a short 
factual answer? 

• Can the response be characterized as detailed? 



Survey Three 



RESPONSE RECEIVED 

• Components of the response. 

• Did the expert answer the question asked? 

• If the question solicited only a factual answer, was the 
response to this question correct? 

• Considering your experience with this site AND this 
expert, would you use this service again? 

• On a scale of one to five, rate this response for 
helpfulness. 

• Was there anything lacking in the response to this 
question? 
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And then \ 


we... 




• Developed guidelines for asking questions, 
including a timeline by which questions would 
be asked and responses would be received. 

• Completed submission of all questions within 3 
weeks. 

• Each submission was given at least 2 weeks 
response time. 
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Next, we.. 






• Established willing participants among the sites 
we had chosen; 0 sites declined participation. 

• Determined pseudonyms from which to ask 
questions, including establishing digital identity 
with free e-mail accounts and/or login names 
and passwords. 
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The Good: 


5 




• Findings are preliminary and not yet exhaustive 

• What we can show you today. . . 

• Site Characteristics 

• Response Rates 

- Overall 

- Sites 

• Analysis of Responses 

~ Overall 

- Sites 

- Question Types 



The Whole Thing Looks Like This... 




Site Characteristics 



• Time to enter site and ask question 

- Overall 4.75m 

- Com 5.4m 

- Non-com 4.1m 



• Method of submitting question 





Average 


Com 


Non-com 


Web Form 


60 


60 


60 


Bulletin 


15 


30 


0 


E-mail 


25 


10 


40 



Site Characteristics 



• FAQs 





Average 


Com 


Non-com 


Yes 


30 


20 


40 


No 


70 


80 


60 



1 commercial site and 3 non-commercial sites 
requested that the user look at FAQs before 
asking their question. 
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Site Characteristics 



• Policies: 





Average 


Com 


Non-com 


Qs not 
answered 


20 


10 


30 


Turnaround 

time 


35 


10 


60 


Information to 
Include 


15 


0 


30 


Homework 

Advice 


20 


10 


30 


Limitations 


35 


10 


60 



Response Rates 



OVERALL 

- Received 267 responses to 168 questions of 240 
asked. 





Average 


Com 


Non-com 


Fact 


75.5 


77 


60 


Source 


67.3 


70 


40 


Scope 


55 


70 


40 


Totals 


70 


73.3 


46.7 
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Response 


Rates 




BY SITE 

• removed out-of-scope questions 

• non-commercial sites combined 



Response Rates by Site 



RANK 


CODE 


RATE 


1 


B 


20/20 


2 


1 


19/20 


3 


F 


18/20 


4 


E 


16/20 


5 


C 


15/20 


5 


G 


15/20 


7 


H 


14/20 


7 


A 


14/20 


9 


ORGS 


11/20 


10 


J 


9/20 


11 


D 


7/20 




1S6 
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Response 


Analysis 




OVERALL 

• 51% of responses answered the question asked; 
another 17% were considered to maybe answer the 
question. 

• The median response time was 3 hours and 3 minutes. 

• 48% of the time we indicated we would use the service 
and expert again; another 21% we indicated maybe. 
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Response 


Analysis 




• Only 12 of the 267 responses asked for more 
information. 

• Answers ranged from 1 word to 1 ,397 words in 
length. The median word count was 29. 

• 69% of the time we indicated that something 
was lacking in the response 
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Response 


Analysis 




BY SITE 

• excludes out-of-scope questions 

• collapses non-commercial sites 

• nine measures 



Site Rankings 



Requested Additional 1 
Information j 

Rank Code Rate j 


1 


H 


20.0 


2 


D 


14.3 


3 


’ A 


6.7 


3 


C 


6.7 


4 


B 


4.1 


5 


1 


4.0 


5 


F 


4.0 


7 


G 


3.8 


8 


E 


0 


8 


J 


0 


8 


ORGS 


0 



Responses per Question 


Rank 


Code 


Number 


1 


B 


7.35 


2 


J 


1.89 


3 


G 


1.73 


4 


E 


1.50 


5 


1 1.32 


6 


F 


1.11 


7 


A 


1.07 


7 


H 


1.07 


9 


D 


1.00 


9 


C 


1.00 


9 


ORGS 


1.00 



0 

ERIC 
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Site Rankings - Response Time 



RANK 


CODE 


MEDIAN 


1 


C 


1.2 h 


2 


A 


3.3 h 


3 


1 


4.0 h 


4 


B 


6.8 h 


5 


D 


8.5 h 


6 


F 


13.9 h 


7 


E 


16.4 h 


8 


ORGS 


1 day, 4.5 h 


9 


H 


1 day, 18 h 


10 


G 


5 days, 1 .4 h 


11 


J 


6 days 



Site Rankings - Word Count 



RANK 


CODE 


MEDIAN 


1 


H 


105 


2 


ORGS 


81 


3 


F 


71 


4 


C 


55 


5 


1 


54 


6 


J 


41 


7 


D 


33 


8 


G 


32 


9 


A 


28 


10 


B 


27 


11 


E 


23 



Site Rankings - Answer Included 



RANK 


CODE 


RATE 


1 


ORGS 


.25 


2 


A 


.23 


3 


1 


.19 


4 


J 


.18 


5 


C 


.17 


5 


M 


.17 


7 


D 


.14 


8 


B 


.13 


9 


G 


.12 


10 


E 


.11 


11 


F 


.08 



Site Rankings - Answered Q Asked 



RANK 


CODE 


Y/M 


1 


1 


92.0 


2 


E 


91.7 


3 


ORGS 


90.9 


4 


F 


70.0 


5 


G 


69.3 


6 


B 


61.8 


7 


A 


60.0 


8 


J 


58.8 


9 


H 


53.4 


10 


C 


46.7 


11 


D 


42.9 
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Site Rankings - Something Lacking 



RANK 


CODE 


Y/M 


1 


D 


100 


2 


C 


83.6 


3 


B 


76.9 


4 


E 


75.0 


5 


A 


73.3 


6 


G 


69.2 


7 


H 


66.7 


8 


ORGS 


64.0 


9 


F 


60.0 


9 


1 


60.0 


11 


J 


52.9 



Site Rankings - Helpfulness 



RANK 


CODE 


MEAN 


1 


1 


3.56 


2 


F 


3.20 


2 


H 


3.20 


4 


A 


3.13 


5 


E 


3.08 


6 


ORGS 


3.00 


7 


G 


2.81 


8 


B 


2.76 


9 


J 


2.65 


10 


D 


2.29 


11 


C 


2.27 



Site Rankings - Use Again 



RANK 


CODE 


Y/M 


1 


1 


96.0 


2 


A 


93.3 


3 


ORGS 


81.8 


4 


E 


75.0 


5 


H 


73.3 


6 


F 


70.0 


7 


G 


69.2 


8 


C 


66.6 


8 


B 


66.6 


10 


J 


47.1 


11 


D 


28.6 



Response Analysis 



BY QUESTION TYPE 





Median 

Response 

Time 


Median 

Word 

Count 


Fact 


6 hours, 26 
minutes 


29 


Source 


18 hours 


31.5 
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Was the Answer Verifiable? 

FACT QUESTIONS ONLY 



Yes 


50.4% 


Maybe 


18.7% 


No 


30.9% 





/ 

Conclusio 


ns 




• Generally, not so bad 

- response rate at 70% 

- serious answers 

- over 50% verifiable 

• Subjective Measures 

- researchers would use again even though answers 
almost always lacking 





(— 

Response 


Analysis 




BY SUBJECT 

• Barely begun 

• Education, Art, and Health received the fewest 
responses. 




? 0 0 





/ 

Implicatio 


ns 




What about the 55% rule? 

- The studies are not exactly parallel 

- Ability to pass on questions 

- Different kind of questions 





Implicatio 


ns 




Is the reluctance to answer source questions 
online justified? 

- Word count 

- Quality of response 

- Lack of reference interview 




Encouraging Online Questioners to Question Their Questions 



Tim Steury 

Director, Ask Dr. Universe, Washington State University 

Ask Dr. Universe, a question and answer Web site ('www.wsu.edu/DrUniverse') sponsored by 
Washington State University, has reached an impasse. Traffic to the site has become so heavy 
that it has become impossible to answer all the questions. This has presented a quandary in that 
further promotion and development of the site is impossible without drawing even more questions 
that will go unanswered. 

In response, we are re-focusing the site. First, we are developing reference links to re-direct the 
purely informational questions and routine homework questions. We are also more pointedly 
challenging users of the service to consult with their local librarians. This screening process will 
address approximately half the questions received daily. The rest are more problematic, as they 
are still far more than we can possibly answer. And they are not easily or routinely answered. 

Our approach to these questions will be to answer the best ones — meaning the most thoughtful 
and original. This is not as callous, or elitist, as it first sounds. We are developing a transparent 
online tutorial to encourage our questioners to consider their questions more carefully. In other 
words, we are looking at the “good question” as the most important part of critical thinking and 
the learning process, because it helps the questioner formulate thought-provoking ideas of his or 
her own. 

Oddly, this idea has seldom been addressed systematically by educators. Despite a revered 
tradition of excellent teachers/questioners, starting with Socrates, the Ask Dr. Universe team has 
not been able to find extensive pedagogical material that focuses on the question. 

We have begun weekly discussions among a group of scientists, philosophers, librarians, and 
other types addressing the question, “What is a good question?” Already the discussion has been 
lively and the perspectives varied. 

What we hope emerges is not only a broad categorization of the Question, but also a qualitative 
and probably somewhat subjective analysis of the Question. We plan on our philosophers keeping 
us honest. Already, what the theoretical physicist in our group considers a “good question” is 
different from what the educational theorist calls a “good question.” Already, the criteria range 
from the epistemological to the aesthetic. 

At the end of our discussions, we will attempt to incorporate our collective analysis into an 
engaging challenge that will meet each questioner who visits our site. We aim to synthesize 
criteria that will reveal commonalities between a question asked by an inquisitive five-year-old 
and one asked by a creative scientist. To further encourage our readers to question their questions, 
we will have a “best question of the week” contest. The winner will receive a new dictionary or 
Dr. Universe T-shirt or other suitable prize. But most important, Dr. Universe will explain WHY 
that question was picked as the best. 

Of course the best questions will be answered, often in our widely distributed newspaper column. 
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E-mail Service Centers at the U.S. Department of Education: A Study 

Joanne Silverstein 
Information Institute of Syracuse 

Presentation 

Introduction 

For several years, the United States Department of Education has offered digital 
reference services that are well established and committed to customer service. Last year, 
the department conducted a major research initiative to optimize those services. This 
presentation discusses the research methods and results. 
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The Importance of Digital Reference in 
Supporting Critical Thinking in Distance Education 

Joseph A. Meloche 

School of Information Studies, Charles Sturt University 
Abstract 

This paper will discuss a range of issues relating to digital reference in support of distance 
education (DE). The development of a tailored digital reference service can be one of the 
key factors in a successful program of study in DE. This paper examines the increasing 
emphasis in DE for students to become independent learners with critical thinking skills, 
and the supporting role of libraries and reference services in that process. 

The paper will address the following issues: the enhanced role of reference services in 
distance education programs and flexible learning initiatives, education for the provision of 
digital reference, the importance of supporting critical thinking strategies and fostering 
independent learning, and strategies for increasing cooperative ventures between 
academics and academic librarians. 

The Australian Educational Context 

The situation in the Australian academic educational environment has been directed by 
changes in government policy that provide support to higher education. Kemp (1999) 
identified that graduates should have knowledge and thinking skills as outlined in Table 1: 



Table L Knowledge and Thinking Skills 



Knowledge Skills 


Thinking Skills 


• Have an appropriate level of literacy and 
numeracy skills 

• Be able to identify, access, organize and 
communicate knowledge in both written and 
oral English 

• Have good listening skills 

• Have an international awareness 

• Have the ability to use appropriate technology 
to further the above 


• Be willing to challenge current knowledge and 
thinking 

• Have conceptual skills 

• Have problem-solving skills 

• Be creative and imaginative thinkers 

• Be able to combine theory and practice 

• Be able to reflect on and evaluate their own 
performance 



The components of the “Thinking Skills” section are almost synonymous with the key 
attributes of critical thinking, and they point to the reality of students’ experiences. 
Students in Australia, as in many other parts of the world, are faced with an over- 
abundance of information, and they face the responsibility of critically assessing the worth 
of the information they find. Changes in the delivery of reference services will be most 
successful if they remain flexible and responsive to the model of lifelong learning. 

A Question-Based Approach 

A central problem with critical thinking is that it appears to be an abstract idea applied 
roughly across a range of subjects. In the worst cases, students are told to think critically 
after having been hand fed material throughout the semester. Students who have merely 
read supplied materials are not likely in a position to critically assess them. The provision 
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of materials by an instructor may imply that such materials are the best or most 
appropriate, unless specifically offered as a worst-case example. While academics may 
supply material, it should be presented as an example of a more extensive collection. It is 
helpful to give the students a taste of a particular subject by suggesting specific resources, 
but it is critical that students make the effort to select, evaluate and assess the material they 
use in order to explore the subject further. In this way, critical thinking can be incorporated 
from the beginning of the students’ research. 

The following is a brief example of how this may take form: 

1 . Learning Task: Provide an overview of the subject and list learning expectations 
for students. 

2. Key Questions: Provide extensive questions, some contained within sections and 
supported by brief examples. 

3. Expected Outcomes: Link outcomes to the schedule and/or subsets of the subject. 

4. Subject Key To Literature: Cite subject headings that help students locate 
material germane to their study. 

5. Authors: Include a selected bibliography as a starting point for research. 

6. Date Range, etc.: Supply helpful subject-related information, such as historical 
dates and geographic touchstones. 

7. Criteria for Assessment: List criteria for assessment at each level, and for each 
assignment; the assessment should leave no doubt about what is required to achieve 
the various levels of success for each assessable item (Meloche, 2000). 

The Enhanced Role of Reference Services 

Reference services for DE students have traditionally been very supportive and proactive. 
Information specialists have worked closely with academic departments and subject 
coordinators to provide students with sufficient material to successfully complete their 
subjects. This allows information specialists to be precise in what they supply and 
informed about what is required. 

The difference in taking a critical thinking approach is that the emphasis moves from the 
academic’s requirements to the student’s requirements, and the student’s requirements vary 
greatly from student to student. The critical thinking model encourages students to define 
their information needs and assess the information they locate. The students adopt an 
active information literate approach to their learning. 

Reference services, however, especially a distance education or virtual service, must be 
prepared to supply the resources and services that students require regardless of their 
geographic location. It is critical that such services be interactive with and responsive to 
the students. 

Charles Sturt University (CSU) offers this type of reference service to its students. CSU is 
a multi-campus university with 19,473 DE students, as compared with 6,587 on-campus 
students (Charles Sturt University, 1999). Given this large DE population, which includes 
overseas students, delivery of reference services has posed a number of challenges to 
university staff. Flexible or alternative modes of delivery have been investigated. 
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Education for the Provision of Digital Reference 

Education for the provision of digital reference occurs on two levels: education provided to 
students who are studying to become librarians and training provided by working reference 
librarians to students who use digital reference services. 

In educating students in librarianship we seek to expose them to the benefits and the 
difficulties that are associated with remote or digital reference and to become increasingly 
aware of the processes that are involved in information seeking. Learning is structured so 
that students first read widely to develop a sense of the issues and problems that are being 
studied and argued, and then develop a situation or problem-based scenario that can be 
addressed. 

Once such a problem scenario has been developed, the students then think through the 
possible approaches to take to resolve the various aspects of the problem and the types of 
resources and services that would assist in its resolution. In addressing this scenario, they 
also consider the audience or individuals concerned, the education and facilities that they 
can likely access, the suitability of the available systems, and services to which they will 
have access. 

The purpose of this learning task is to get the students to work through a specific problem 
from the perspective of a specific group of users. Thus, they are expected to record the 
processes that they go through in addressing the problem and the considerations they make 
as they begin to construct a solution. It is expected that the solution will include a package 
of services and resources that cover all facets of the problem. By the time the students 
begin to evaluate specific resources or services, they are able to do so with an 
understanding of their limitations in education, experience, time, and financial resources. 

The training provided by working reference librarians to students who use digital reference 
services is developing and it varies depending on the situation. Those universities that have 
residential schools for DE students still focus on in-house training and tours of the physical 
and electronic resources. This type of training, while unquestionably welcome by the 
students, may be of limited value when students return to their remote locations. It has the 
benefit of introducing the students to the professionals and the array of resources and 
services. However, this approach does not demonstrate the environment in which they will 
study and work. Alternatively, universities that do not have residential schools need to 
develop virtual training that will at least be appropriate for the remote students, and 
includes on-line tutorials, instructional Web resources, and on-line or interactive help 
services. Electronic communication supplements information resources and aids, and will 
greatly affect the delivery of reference services in the future. 

Virtual reference service to remote or DE students presents a number of opportunities and 
challenges. Advantages include the development of resources and activities that foster a 
learning community among DE students. It is important to provide dedicated professionals 
and services in order to effectively utilize technological solutions (e.g., MOOs, Net 
Meeting, Forums, etc.) that can add value to the service. 

Supporting Critical Thinking Strategies and Fostering Independent Learning 



> 



Critical thinking approaches require an enormous amount of support services and 
resources. It is a common misconception that strategies requiring students to think 
independently and critically will free the time of educator and information professionals; 
experience has shown the opposite is true. As opportunities increase for students to work 
independently and critically, the need for assistance increases accordingly. Thus if 
measures are undertaken to use the “question based approach,” resources and strategies to 
support the students must be in place. 

To successfully achieve this approach, the instructor, in consultation with the librarian, 
needs to identify the students in a specific course of study. The librarian, like the students, 
should be supplied with the subject material. It is critical that the librarian be aware of or at 
least have ready access to information such as deadlines and requirements for the course. It 
is also important that support for remote or DE students is not seen as a lower priority than 
support for on campus students. The level of need for the distance education student is 
greater; this should be reflected in the level of support for these students. This has not been 
the case at CSU and possibly at other universities providing distance education. Internal 
students are sometimes viewed as “real” students and DE students are considered less of a 
priority. Traditionally, many library services, even computerized, have been only 
accessible locally. 

Many barriers confront distance education and complementary library services including: 

• Access to facilities: Computer labs, printing facilities, local databases, local 
CD-ROM services, microfiche and multi-media collections are unavailable to 
distance education students. 

• Access to support: Reference desk, lab support, and advisory staff frequently 
work in a location adjacent to the collections or equipment. 

• Access to information: Web pages cannot compare with the support of a well- 
staffed reference department that exists in addition to a wide range of reference 
and advisory resources in a variety of print and electronic formats. 

While it may be possible for DE students to have their own equipment, and for libraries to 
provide DE or remote reference staff, few do. It should be remembered that all of the 
above services are expected for internal students and yet DE students need greater levels of 
services and support. While it may not be suitable or even desirable to duplicate the 
services provided for DE students, it is essential that their needs be met. 

Research is required to examine the perceptions of DE students in regard to their use of 
remote or virtual reference services. Too often we have simply tried to duplicate existing 
services for internal students or to teach DE students to use services that we introduce, 
without the benefit of research into how students understand information, information 
technology and information seeking. Students’ needs should be first considered during the 
design stage of services, not the delivery stage. 

Strategies for Increasing Cooperative Ventures 

One strategy for collaboration between academics and academic librarians is research. The 
research is necessary and should involve the full range of participants, from librarians to 
academics to instructional designers, and most importantly, students. The research must be 
capable of exploring how systems needs are conceptualized by the different participants in 
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the chain of use (Meloche, 1999), and all should participate in the development of 
workable strategies. 

The problems and issues associated with the increased use of remote and virtual reference 
cannot be solved after implementation of services has occurred. Students should not be 
asked to evaluate only the services that currently exist. They need to be asked what they 
think should exist and how they might use it. They need to be participants or subjects in 
research that will affect how information services will evolve. 
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Assessing Users’ Needs 



Pauline Lynch Shostack 
AskERIC 

Presentation 

Introduction 

AskERIC responds to over 40,000 questions a year. The questions cover a 
multitude of subject areas and come from a wide range of users. Responses are sent 
within two business days to requests received at AskERIC. Requests are received via our 
Web form (http://www.ask.org/Qa/userform.html) and from direct e-mail. AskERIC 
prides itself on providing education information with the personal touch. To accomplish 
this goal, user studies and data collection are constantly conducted. The findings from 
several of these efforts have led to changes and enhancements to our services and 
resources. 

This year, AskERIC conducted three different studies to find out even more about 
our users. In particular, the studies provide insight into the topics users are interested in, 
geographic location of users, and satisfaction level with responses received. A faculty 
member at the University of Washington, Seattle is conducting research to compare 
responses across different types of AskA services. 

Additionally, doctoral students and faculty occasionally conduct outside research 
on various aspects of service. In 1999, a doctoral student at Syracuse University 
completed a dissertation on information needs, using AskERIC users as subjects. 

This report summarizes the various methods that AskERIC has used to obtain 
information about users as well as results. Below is a list of the user studies and data 
collection methods that are discussed in this presentation: 

■ Surveys - conducted in 1998 and 2000 

■ Focus group - conducted in lanuary 2000 with a local group (Syracuse, NY) from 
the education community 

■ Data collection from the service’s question submission form - data collected from 
lanuary 1 to the present 

■ Outside research efforts 

o Dissertation - PhD Student at Syracuse University conducted study for 
dissertation in the Fall of 1999, and another student conducted a study in 
1997 on the service 

o Faculty Research - Comparison of responses across AskA Services is 
currently being conducted 
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Overview 



■ Methods 
< Surveys 
» Focus Groups 
> Question Submission Form 
» Subject Line Analysis 
» Outside Research Efforts 
• Dissertations 
. Faculty Research 

« Results 
a Action taken 
a Future directions 



^ Survey Data Collection Details 




Survey ‘98 


Survey ‘00 


Method of 
Data 

Collection 


E-mait Survey 


E-mail Survey 


Time Frame 


Feb- Mar 1998 


Feb - Mar 2000 


Pool of 
respondents 


689 


443 


Number of 

actual 

responses 


196 


79 


Response 

Rate 


28.4% 


17.8% 



1998 & 2000 



a The design methodologies for both 
surveys were fundamentally the same. 
■ The majority of changes were in the 
question format and wording. 
b The cover letter was also enhanced to 
provide more options for submission 
and other details. 
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— k 2000 Survey Timeline 




^2000 Sampling Procedures 


■Jt 

■ Refine survey documentation and 




Jfc 

■ Surveys were sent to every third user of 


design questionnaire (summer) 




the AskERlC Service via e-mail 3 days 


■ Pretest Survey (September) 




after they received a response. 


■ OMB Approval (November- December) 




d A script was created to pull the e-mail 


■ Administer survey (January) 




address for each survey recipient from 
the carbon copy accounts of each 


■ Analyze data (March) 




clearinghouse. 


■ Report Findings (Late March/April) 







J-oL 2000 Instrumentation 




- 2 ^, 2000 Instrumentation 


"Jt 

- Cover letter (enhanced since 1998) 




■ Self-administered electronic survey 


• Subject line: AskERlC Survey: Please help us serve 




• Questions based on feedback from the 1998 


you better! 




AskERlC survey & pretest 


■ Introduction to survey 

• Approximate time to complete survey (iomins) 

■ Submission options (e-mail, web, postal) 

• Deadline (2/29/00) 

• Result information (e-mail required) 

■ OMB Paperwork Burden Statement 




■ Instructions at the beginning and throughout 

• E-mail & Web versions 

• Several types of questions 

. Likert Scale 
. Yes/ No 
. Open Ended 
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^-L 2000 Reporting 




E-mail Survey Pros/Cons 


”®P*L 

o Complete Report 




a Ability to reach a large number of users 


> Audience (ERIC Staff, US Department of 
Education) 

* Detailed description of the survey design 




and possibly non-users 
= Low response rate to surveys 


■ In-depth analysis of results 




a Tendency to leave open-ended 


b Executive Summary 




questions blank 


■ Audience (Survey respondents, AskERIC Web site 




a Users are either extremely happy or 


visitors) 

■ General overview of purpose of the survey 




extremely dissatisfied 


■ Summary of relevant findings 

■ Statement about how confidentiality was insured 




a Relatively low cost 



-^AskERIC Local Focus Group 




-^AskERIC Focus Group 


Who attended? 




We asked... 


■ Librarians (public and school) 

■ Teachers (public and private schools) 




■ How do you use the Internet in relation to your 
educational needs? 


■ Home schooling parent 

■ Local Commissioner of Education 

■ Secondary school administrator 




■ Which Web sites, discussion groups, QA Services or 
reference services do you find the most useful? 

« What comments do you have about the AskERIC Site, 
AskERIC Q&A Service, and ERIC Database via 
AskERIC? 






> If you have never used AskERIC (or particular 
features of AskERIC), why not? 
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-2L AskERIC Focus Group 




-2^ Focus Group Pros/Cons 


Feedback we received... 




JL 

• Groups comprised of local attendees make it 






difficult to generalize findings. 


- Provide full text of ERIC documents. 




■ Reach the non-user of your service. 


■ Include a section that explains the routing of a 
question and outlines the Q&A process. 




■ Can be expensive. 


■ Create a public archive of responses. 




« Detailed feedback but difficult to summarize 


- Explain more clearly the ERIC Clearinghouse System, 
especially the clearinghouse subject specifications. 

. Provide an online place for teachers to communicate 




comments/suggestions objectively. 

■ Group members can be influenced by others 
in the group. 


with other teachers/school districts. 





Question Submission Form 




-2^- Demographic Statistics 


“ape “ 

Questions asked on the form 




Would you Uka our response to address any specific aducational lavai(s)? 
Data collected 1/1/00 - 9/30/00 
Adul Brfuc.lon 


« Would you like our response to address any 
specific educational level? 




B»rfc> Chilhood 
12% 




« How do you plan to use this information? 








a In what capacity are you asking this 
question? 




|{*hcr Wuc.lraA / 

U% \ / 


mmm - 

^|ii3l SchooWunnr 
H*h 


a What state and/or country are you writing 
from? 




lt*h Sch 
26% 


oo) 
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International Statistics 



Question* received from over 151 countries since January 2000 
Most questions received from the following countries: 



Canada 


676 


Australia 


502 


United Kingdom 


398 


Israel 


148 


Now Zealand 


125 


Malaysia 


124 


Hong Kong 


117 


India 


110 



Data coOectBd 1/1/00 - 9/30/00 
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^ Submission Form Pros/Cons 




-^Subject Line Analysis 


■BL 

■ Multiple submissions of the same 
question can skew results. 

■ Users that select all choices for 
questions. 








■ Simple method to reach a large 
percentage of users. 














5076 Subject Lines 
,H March 2000 



■XL Subject Line Pros/Cons 




Dissertation 


■ We are now able to answer the 




JE. 

Use of Human Intermediation in 


question, "What types of questions do 




Information Problem Solving: 


you receive most often?" 




A User's Perspective 


■ Subject line may not reflect total scope 




By Makiko Miwa 


of question. 






■ Not everyone is changing subject lines 




Goal: to increase our understanding of the 
purposes and situations for which users request 


to reflect question topic. 




various tasks of human intermediaries. 







Dissertation - 

Why use AskERIC? 



Why AskERIC? 



a Need help searching the ERIC database 
d Found AskERIC while doing an Internet 
search and became hopeful 
> Want to verify own search 
process/results 

a Past success with AskERIC or ERIC 



-Cl Dissertation - Satisfaction 




Dissertation - User Goal 


■p. 

Users are generally satisfied with 




• Degree-seeking Goal - looking for anything 


AskERIC and appreciate the following: 




topical 

. Evaluated AskERIC more positively overall 

• Decision/Action Planning Goal - want facts, 


a Easy to use 




expert opinion, and research supporting their 
point of view 


a Quality of information provided 




. Evaluated AskERIC more critically for noise reduction & 
adaptability 


a Time saving 




• Teaching Goal - want lesson plans or 


a Quantity of information provided 




practical resources, not research 

. Evaluated AskERIC less positive for time saving 




o o ^ 

^ A O 



Dissertation - 




Dissertation - 


-^L Recommendations 




Recommendations 


•m 

Researcher Recommendations for AskERIC: 




Researcher Recommendations for ERIC: 


(based on user suggestions) 




(based on user suggestions) 


. Adjust responses to reflect user's situation 
. Clarify information needs when necessary 
- Update links to Internet resources periodically 
. Provide links to other AskA services that can directly 
answer the question 

. Determine the user's level of information problem solving 
(Q&A Form) 

. Provide clearer explanations of how to obtain full text 




. Develop digital ERIC documents with links from citation 
. Provide links to e-journals from citation 
. Provide terse conclusions in ERIC abstracts 
. Provide ERIC Digests for major educational theories and 
models (e.g., Bloom's Taxonomy) 

. Include detailed copyright information for ERIC 
documents 

a # of copies permitted for educational use; permission to 

reproduce does not apply to third parties; procedure to obtain 
permission to make additional copies 



Faculty Research 




^ Research Pros/Cons 






Jit 


■ Faculty at the University of Washington 




■ Outside perspective and angle can be a 


worked on a project that involved 




plus. 


reviewing responses sent from various 




■ Focus is usually not on the service 


AskA services. 




itself. Findings pertaining to the service 


■ The project has just concluded and 




are secondary. 


results and a final report are pending. 




■ Phone survey responses are difficult to 






summarize objectively. 
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Results - Common Issues 




Results - Common 
-2^ Compliments 




■ Full-text concerns m 

• Discussion groups continue to be 
ranked lower than other resources <»»«) 

■ Inability to understand codes in 
citations m 

■ Response did not fully address question 

(dissertations, surveys) 

■ Process not fully understood m 




■Pi- 

■ Quick response (dissertation, surveys) 

■ Information was on target (dissertation, surveys) 

a Appreciate large quantity of information 

(dissertation, survey) 

m Saved users time (dissertation, survey 98 ) 




Results - Demographics 



Elementary Education Scope 



. Over 25% for both surveys and question 
submission form 



■ K-12 Teachers/Post-Secondary Students 

■ Make up almost 60% of respondents in 
most data collection methods 

■ New/Repeat Users 

■ 50% for each category in both surveys 




Action Taken Based on User 
Feedback 



Redesign of QA form 



■ ERIC Database enhancement on 
AskERIC site 



■ Creation of specialized letterheads 

■ Addition of response archive to QA 
portion of AskERIC Site 
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Future Directions 



■ Survey 2001 

• Continue e-mail survey 

• Create short Web surveys to evaluate Web 
site and reach some of our non-users 

■ Revise question form to obtain more 
information from our users 

■ ERIC evaluation being conducted within 
the next several months 



O 

ERIC 
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User Situations in Digital Reference Service: 

An Evaluation of the AskERIC Q&A Service 

Makiko Miwa 

Epoch Research Corporation 

Presentation 

Introduction 

This research addressed the situations in and the reasons for which people make 
requests of human intermediaries in solving their information problems. This study looked 
at the use of AskERIC, a digital reference service specializing in educational research and 
practice [part of the Educational Resources Information Center (ERIC)], as a case study in 
human intermediation. The study collected data through telephone interviews with 62 
AskERIC clients. Interview data were analyzed using a modified constant comparative 
technique. The findings were synthesized into a conceptual model that captured patterns of 
associations between users’ situational factors and tasks they requested of AskERIC. This 
model provides a framework for a better understanding of users and ways to incorporate 
their needs into future digital reference services and system design. In addition, the study 
developed two taxonomies that are potentially useful for future research and practice. 
“Taxonomy of Tasks Requested of Intermediaries” (see Table 1) has potential for 
identifying and categorizing tasks requested of human intermediaries; “User-Based 
Evaluation Criteria of the AskERIC Q&A Service” (see Table 2) has potential for 
identifying strengths and weaknesses of a variety of digital reference services. 
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VRD Digital Reference Conference 2000 




Outline 


User Situations in Digital Reference Services: 




• Why study human intermediation? 


An Evaluation of the AskERIC Q&A Service 




• Study overview 

• Research questions 


October 16-17, 2000 




• Research design 


by 

Makiko Miwa Ph D. 




• Study limitations 


Miwamaki@aol.com 




• Study findings 

• Implications for AskERIC and other Digital 
Reference Services (DRS) 

• Implications for ERIC 



Why Study Human Intermediation? 

• Weak theories and contradictory findings 
-lack users’ perspective 

- lack categorization of requested tasks 

- lack process approach 

-ignore external (social & environmental) 
situations 

• Users of Internet-based digital reference 
services were not well studied 



Study Overview 

• Goal: to increase our understanding of why and 
in what situations users request certain tasks of 
human intermediaries. 

- take an exploratory approach 

- focus on users’ perspective 

- capture information problem solving (BPS) as a process 

- naturalistic 

- account for dynamically changing internal and external 
situations 

- Invite clients of AskERIC Q&A Service as participants 
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Research Questions 

RQ1 : What kinds of tasks do users request of 
human intermediaries? 

RQ2: What situational factors do users 
perceive to be salient when they make 
requests of human intermediaries? 

RQ3: What patterns of associations, if any, 
are observed between users’ situational 
factors and tasks requested of human 
intermediaries? 



Research Design (3 Phases) 

• Phase 1 

- Developed preliminary conceptual framework (literature 
review) 

- Selected study setting (clients of AskERlC Q&A service) 

- Designed and tested data collection instruments 

• Phase 2 

- Analyzed past requests (developed taxonomy) 

- Interviewed participants by phone 

- Analyzed telephone interview (content analysis) 

• Phase 3 

- Identified patterns of associations among situational factors 

- Modified conceptual framework 

- Reported findings and implications 



Research 

Clients 

AskERlC 



Temporal Sequence of Telephone 
Interview Procedure 

I ir 1 - I : s r!, lniCT ' i ' w 

6 o- oo 



Receive Receive 
Receive Recmitmcnt 



- Q O O- 

2 business days j 



{ 3 business days I 



© 



Within 2 Weeks 



Time 



Study Limitations 

• Capture salient and memorable situational 
variables 

• Not generalizable to the population of 
AskERlC users 

- purposeful sampling 

- did not recruit minors and non-US clients 

- self-selection bias 
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Study Findings 1/9 

• Multiple levels of User Goals 

- IPS goals: What participants intended to 
accomplish through the IPS process 

- Goals of using AskERIC: What participants 
intended to accomplish through the use of 
AskERIC 

- Tasks requested: What participants requested of 
AskERIC 



Multiple Levels of User Goals 
(^^IPS goal^^ 



“~T. 






asks requested of 
AskERIC 



Explanatory Note 

Associations with 
Situational Factors 
Distal goals generate 
sub-goals 

Users’ movement 



Study Findings 2/9 

• Associations between user goals and tasks 
requested 

- When users’ goals were to find relevant information or 
sources of information, their requests are consistent 
with all or part of the goals. 

- When users’ goals were to define information needs, 
(e.g., decide topic of a paper) their requests do not 
reflect the goals explicitly. 

- When users’ goals were to evaluate their own search, 
their requests do not reflect the goals explicitly. 



Study Findings 3/9 

• Users’ situational variables associated with their 
selection of AskERIC 

- Self-searching before sending requests 

• Self-searching failure 

• Encountered AskERIC & opportunistic hope 

• Verification of own search process/results 

- Experiences using AskERIC & ERIC 

» Users who had experience using AskERIC had more elaborated 
mental model of AskERIC and attachment with it 

— Perceived level of IPS skill 

• Users with a low level of perceived IPS skill require help 
throughout IPS processes 
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Study Findings 4/9 

• Evaluation and Satisfaction 
- Very high evaluation score 



Evaluation Score (N=58) 

Mean = 8.3; Median = 9.5; SD = 2.36 
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Study Findings 4/9 

• Evaluation and Satisfaction 

- Very high evaluation score 

- Very high level of satisfaction 



Satisfaction Score (N=58) 

Mean = 8.4; Median = 10.0; SD = 2.2 



Evaluation 

Score 


Frequency 


5 10 15 20 25 30 


1.0- 1.9 


i 


































































2.0 -2.9 


1 


































































3.0 -3.9 


0 


































































4.0- 4.9 


0 


































































5. 0-5. 9 


7 


































































6.0- 6.9 


l 


































































7.0 -7.9 


5 


































































8.0 -8.9 


9 


































































9.0 -9.9 


5 


































































10.0 


30 



































































o 

ERIC 



233 



Study Findings 4/9 

• Evaluation and Satisfaction 

- Very high evaluation score 

- Very high level of satisfaction 

- A high level of agreement between evaluation 
and satisfaction scores (r=. 80 ) 



Evaluation Criteria (See Table 2) 

Easiness (easy to use, make IPS easier; physical accessibility; 
formatting) 

Noise reduction (linkage; online linkage to full text materials; 
selectivity of information; type of information; search skill of 
intermediary) 

Quality of information (comprehensiveness; correctness; depth; 
currency) 

Adaptability (answer to the question; usefulness; interactivity; clarity 
of information; understand requests) 

Time savings (response speed; time-saving in IPS) 

Cost savings (cost-saving in IPS) 

Quantity (as much as wanted; too much; too little) 

DRS features (acknowledgment; humanness; responsiveness; tone of 
message) 

User situations (wording of request messages; frustration level; 
novelty; comprehend response; willingness to read) 



Study Findings 5/9 




Study Findings 6/9 


• Users appreciate AskERIC for: 






- Easiness 




• Users’ situations associated with evaluation of 


- Quality of information provided 




AskERIC responses 


- Time savings 




- wording of request messages 


- Quantity of information provided 




- frustration level 


- DRS features 




- novelty of information obtained 


• acknowledgment 




— comprehension of responses 


• humanness 




• responded 




- willingness to read 


• tone of messages 
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Study Findings 7/9 

• Situational Variables Associated with 
Evaluation (1) 

- Self-searching before sending requests 



• Evaluate information provided by AskERlC by 
comparing it with their own search process/results 



Evaluation Score & Search before AskERlC (N=58) 

Did Search: Vlean~8.3: Media n-f); SD~2.31 

No Search: Mean=8.5; Median=10; SD=2.61 
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Study Findings 7/9 

• Situational Variables Associated with 
Evaluation (1) 

- Self-searching before sending requests 

• Evaluate information provided by AskERlC by 
comparing it with their own search process/results 

- Experience using AskERlC 

• Evaluate information provided by AskERlC by 
comparing it with past positive experience 



Evaluation Score & Experience in AskERlC (N=58) 

Had Experience; Mea»~7.4: Median- 1 #: Sl>~2.8 ! 

No Experience: Mean=8.7; Median=10; SD=2.08 



Evaluation 

Score 


Experience 










Yes 


No 








5 




10 


15 


20 


25 


30 


1.0-1.9 


1 


0 


Ml 




























































2.0 - 2.9 


1 


2 






























































3.0 -3.9 


0 


0 








j 


























































4.0 -4.9 


0 


0 


































































5.0 -5.9 


2 


2 
































































6.0 -6.9 


1 


0 


































































7.0 - 7.9 


1 


6 






























































8.0 -8.9 


4 


4 












■ 
















































9.0 - 9.9 


2 


4 






m 


m 
















































10.0 


5 


23 






nr 


HajH - 








Total 


17 


41 






235 



Study Findings 7/9 

• Situational Variables Associated with 
Evaluation (1) 

- Self-searching before sending requests 

• Evaluate information provided by AskERIC by comparing it 
with their own search process/results 

- Experience in using AskERIC 

• Evaluate information provided by AskERIC by comparing it 
with past positive experience 

- Perceived level of IPS skill 

• Users with a low level of perceived IPS skill evaluate 
responses provided by AskERIC more positively than those 
with a higher level of perceived IPS skill 



Evaluation Score & Perceived Level of fPS Skill (N=58) 
High: Mean-7.8; iVleilimiNi.5: SO-2.71 
Medium: Mean=S.3; Median-10; SD=2.35 
Low: Mean=9.6; Median=10; SD=0.74 
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Study Findings 8/9 

• Situational Variables Associated with 
Evaluation (2) 

- Users’ IPS goals 

• Degree-seeking (n=28) 

• Decision/action-planning (n=20) 

• Teaching (n=7) 

• Others (n=7) 



Evaluation Score & IPS Goal (N=58) 

Degree-seeking: Mean-9. 1 ; Median-10; 80*1.35 
Dccision/aclion-planning: Mean=7.6; Mcdian=8.5; SD-2.85 
Teaching: Mean=7.6; Median=8; SD=3.05 
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Study Findings 8/9 

• Situational Variables Associated with Evaluation (2) 

- Users’ IPS goals 

• Participants with degree- seeking goal tended to 
evaluate information provided by AskERIC more 
positively compared to others 

• Participants with decision/action-planning goal 
tended to be more critical in evaluating information 
provided by AskERIC for noise reduction and 
adaptability 

• Participants with teaching goal tended to be less 
positive in evaluating responses of AskERIC for 
time-savings 



Study Findings 9/9 

• Request messages are short and ambiguous 

• Users’ information needs may be biased due 
to “pretended rationality” 

• Users assume digital documents are 
available on the Internet 

• Users have difficulty locating ERIC Digests 
(special reports on Web site) 



Associations among User Goals and Situational Factors 

Sltuuttmaal Fttort 




Implications for AskERIC and other DRSs 

• Maintain highly appreciated features 

• Useful situational variables 

- user goals (IPS goals & goals of using AskERIC) 

- type of information sought (form/content) 

- self-searching before sending requests 

- experience of using AskERIC 

- perceived level of IPS skill 

• Provide direct answers to specific questions 
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Implications for ERIC 

• Provide online links from metadata to 
digital fulltext on the Internet 

• Make fulltext ERIC documents available on 
the Internet 

• Make it easier to specify type of 
information 

• Make it easier to identify empirical research 
that support users’ contentions 

• Make it easier to find ERIC Digests 
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Information Seeking and User-Intermediary Interactions: 
Informing the Design of Digital Reference Services 



Carol A. Hert 

United States Bureau of Labor Statistics & Syracuse University 

Abstract 

Understanding how users seek information on Web sites and interact with intermediaries 1 can 
inform the design of digital reference services. More specifically, the ways that users frame their 
questions and intermediaries conduct reference interviews can provide valuable guidance for 
designing and supporting these functions in digital reference services. This paper reports on 
research on the users of statistical information and their interactions with intermediaries and 
translates findings into design guidelines. Data-gathering methodology is also discussed. 

Introduction 

This paper reports on investigations that were part of a project designed to understand and 
support user information seeking on Web sites. While Web sites cannot be equated with digital 
reference services, the two share a similar goal~the provision of information and services to a set 
of users. Therefore, what we have learned about certain aspects of user information-seeking 
behavior on these sites is directly relevant to the design of digital reference services. In addition, 
the larger context in which people seek information on a Web site may inform our understanding 
of the design of the particular component, ’’reference service.” 

This study investigated Web sites that provided access to United States Federal Statistical 
Information. More specifically, our investigations focused on the United States Bureau of Labor 
Statistics Web site ( http://www.bls.gov ) and on FedStats ( http://www.fedstats.gov ) which is a 
product of the United States Interagency Council on Statistics designed to facilitate access to 
statistics produced by Federal agencies. They include statistical reports and tabular data, as well as 
tools for the location and manipulation of that information (e.g., search engines, rudimentary 
indexes). In addition, the sites included mailto links (which prompt e-mail messages to designated 
contacts), telephone numbers of various agency helpdesks, and comment forms. This type of Web 
site can be considered a digital library with a digital reference service. Agency personnel 
responded to user questions and managed the mail and telephone inquiries. This structure 
parallels many AskA and digital reference services that distribute questions and coordinate the 
answer process. Our knowledge of user action and interaction on the site has relevance for the 
design of a digital reference service. 

This paper reports on findings in the areas of user questions and uncertainties, and intermediary- 
user interactions. The theoretical basis of this study is the literature of information seeking 
behavior and human computer interaction; the research in this field is summarized as 
“Information Needs and Uses” in the Annual Review of Information Science and Technology. To 
summarize, the literature indicates that: 

• Users experience gaps, anomalous states of knowledge and cannot tell intermediaries 
what they need or want; 
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• Users come to intermediation settings with information needs, as well as affective needs, 
and a repertoire of behaviors; 

• Information seeking is “situational” and “contextual;” each person is in a unique place in 
his or her life. 

These points indicate that to support information seeking, knowledge of users’ tasks and how 
users articulate what they know and do not know may enable us to provide better tools. While not 
generally reflected in the literature on information seeking 2 , intermediaries form an important link 
between users and information. Thus we might expect that knowledge of user interactions would 
provide perspectives on those instances when users are unable to accomplish their tasks. 

The domain of human computer interaction provides theoretical perspectives on the nature of 
human interaction with computing tools. Again, the literature in this area is rich and is impossible 
to summarize succinctly. For this project, two general principles shaped our work. First, user 
action is interaction. To understand how people seek information via a Web site, it was necessary 
to consider the user, the organization, and the Web site as potentially interacting 3 . Second, 
significant insight into user behavior and the implications for design came from looking at 
moments of “breakdown,” moments when users were blocked, confused, or uncertain. The work 
reported here examined moments of “breakdown” that resulted in the submission of questions and 
comments. 



Methodology 

The two theoretical domains and specific principles previously articulated drove the 
methodological choices for the study. To understand user behavior, it was necessary to collect 
data from real users performing real tasks in real situations. We collected data on users’ questions 
using e-mail inquiries, interviews with users, and interviews with intermediaries. We also 
collected data on intermediary-user interaction by observing these interactions in real settings and 
conducting intermediary interviews. The following details indicate our actions and provide 
guidance for those wishing to duplicate them. An effort is made to limit the discussion to aspects 
relevant across settings; however, it is necessary to explain some aspects of the specific setting. 

Collection and Analysis of E-mail Messages 

E-mail messages were analyzed to identify user tasks, questions, problems, and uncertainties. E- 
mail messages sent to the Bureau of Labor Statistics Web site over two months (in 1997) were 
analyzed. There were 569 messages containing 827 questions. Although it is important to keep in 
mind that e-mail requests represent a self-selected sample of the overall user population, the 
volume of requests and variety of topics and user expertise represented make this sample 
valuable. 

Researchers inductively derived a content analytic coding scheme for the e-mail messages. An 
inductive strategy is useful in situations where no taxonomy exists prior to analysis (as was the 
case here). The scheme development process followed general principles provided by 
Krippendorf (1980) and Holsti (1969). As messages were read, categories were preliminarily 
developed. After the analyst had a sense that no new categories were being added to the scheme, 
the preliminary scheme (along with associated coding rules) was formalized. A second analyst 
then received the scheme and both analysts coded the same subset of the messages (10% of the 



sample). Coding decisions were jointly reviewed to confirm that the scheme was detailed enough 
for any coder to reach the same decision about codes for a message. The statistic kappa (Cohen, 
1960) was used to verity that the coding agreement is due to the reliability of the coding scheme 
and not random chance. At the point our kappa was tabulated, our value was .71 4 . 

The coding scheme employed is indicated below (Tables 1 A and IB). Two dimensions were 
coded for each question. In all cases, the correspondent's language was used. The first dimension 
captured the content of the question. The second captured the nature, or strategy of the query. 
Both dimensions are necessary to understand the nature of a particular question. 



Table 1A. E-mail Coding Scheme: Content Dimension 



Dimension Name 


Dimension Definition 


System 


Questions about the Web site itself 


Data 


Questions related to actual values of variables 
or for actual information (e.g., "I need info on 
the economic outlook for Atlanta") 


Methods 


Anything related to how the data were 
collected such as how many surveys were 
conducted 


Metadata 


Information about the meanings of variables, 
codes, etc. 


Publications 


Requests for physical documents 


Costs 


Questions relating to how much things cost 
(such as "how much would it cost me to get 
data on...") 


Tools 


Questions related to data manipulation tools on 
the site 


Other 


All other types of content 



Table IB. Strategy/Question Type Dimension 



Dimension Name 


Dimension Definition 


How 


The process 


When 


The time of 


Where 


Location/access to, including directions to 
answer questions such as "can you direct me 
to..." 


Do you have 


Existence of information/entity 


Is this an error 




Who 




Other 





Using the schemes, messages were coded and numbers of questions tabulated. After determining 
that only a few categories had large numbers of questions associated with them (what/data, 
where/data and do you have/data), a more specific scheme was developed to gain a more detailed 
understanding of the nature of these questions. The scheme that was developed had two 





dimensions. The first is the content dimension: what specific type of statistic is being requested? 
The second captures characteristics of the data requested. Three characteristics were indicated by 
requesters: regionality, time, and amount (or number) of statistics wanted. An analyst coded all 
questions which fell into the three most numerous categories listed above. 

User Interviews 

Telephone interviews were conducted during January-February 1998, with thirteen past users of 
the FedStats Web site. The chair of the FedStats taskforce sent an e-mail message requesting 
participation in a short interview to FedStats users who had submitted an online comment form 
after October 1997 and who had included an e-mail address. Approximately 150 users were 
contacted. If the user indicated willingness to participate in an interview, the e-mail message was 
forwarded to the researcher who attempted to contact the person. Thirteen interviews were 
scheduled and conducted. Respondents were asked to remember the last time they had used 
FedStats and to describe what they had been attempting to find. Additionally, they were asked to 
comment more generally on FedStats features that either helped or hindered them during their 
interactions. Finally, they were asked for any suggestions for improving FedStats. 

These interviews provided useful data. Many users, however, were unable to report 
specifically on their interactions, due to the length of time between their actual system usage and 
the interview. Additionally, the response rate to our request for participation was low and, though 
some users expressed an interest in participating, they did not return e-mail messages or declined 
an interview. 



Intermediary Interviews 

Intermediaries within the agencies and in other settings were also interviewed. A total of 26 
people in nine agencies were interviewed on site in January 1998. A total of 19 intermediaries in 
libraries, schools, and research organizations were also interviewed. Most interviews were 
conducted at the respondent’s workplace, but two were conducted via phone due to logistical 
difficulties. The distribution of intermediaries was as follows: 

• Eight librarians (one from a state library, three from K-12 schools, two from public 
libraries, two from academic libraries) 

• Two K-12 teachers (social sciences, biology) 

• Nine research consultants (in business, agriculture, education, and public health) 
Intermediaries were asked the following five questions: 

1 . What resources/knowledge assist you in answering statistical questions? 

2. What are common types of prerequisite information that users often need to know before 
you are able to answer their questions? 

3. How do you inform them of this information (tell them, refer them elsewhere, etc.)? 

4. When you try to help people narrow the focus of their questions, are there particular 
methods you suggest, such as by geography, by time period, etc? 

5. To what other agency or organization do you most frequently refer users if you cannot 
answer their questions? 



Intermediaries provided useful data about how they assist users and were able to provide 
synthesized understandings of users’ tasks and problems. In a Web environment, it is often 
difficult to contact users as they are geographically distributed. In addition, since Web sites are 
used by a wide range of users, it would be difficult to contact a sufficient number of users to gain 
a broad picture of usage. Given these issues, intermediaries provide the best source of data about 
users. 



Observations of User-Intermediary Interactions 

In addition to interviewing intermediaries about how they helped users, researchers observed five 
intermediaries at their workplaces as they worked with various users. These observations were 
conducted in February and March 1998. Intermediaries were observed for two hours, during 
which time the observer made notes on any and all user interactions (in person or on the 
telephone) that had a statistical component. To maximize the observer’s ability to gather data on 
such interactions, observations were scheduled only with intermediaries in busy settings where 
there was a high likelihood of statistical questions being asked. (For example, the government 
documents section of the local university library was used rather than the public library.) 

The intermediaries observed were: 

• Two reference librarians in the business, science, and technology section of a busy urban 
public library (during two separate observation periods) 

• Two reference librarians in the government documents section of a large academic library 
(during two separate observation periods) 

• A reference librarian in a state library data center 

Informing the Design of Electronic Reference Services 

The data collection and analysis processes previously described generated a wealth of findings 
and design recommendations specific to the domain studied. However, some findings clearly 
extend beyond the domain studied and will be discussed. 

Our first conclusion is that design must be informed by investigation of real users and 
intermediaries. We have found intermediaries to be reasonable proxies for users when it is 
difficult to gain access to users. In addition, examinations of information seeking outside the 
context of a particular service under investigation provides a perspective of user behavior not 
bound by that system. As such, one can see aspects of behavior that may be inhibited by the 
system and/or can better understand how a given behavior on a system fits into the larger context. 

Analysis of user e-mail messages and comments provided a rich picture of users’ questions and 
uncertainties. Our schemes provide one approach to categorization of these comments although 
others are certainly possible. Using our analysis, we were able to pinpoint specific types of 
information requested and redesign the Web site to provide that information in obvious ways. In 
an electronic reference setting, one may begin to build FAQs or other tools for common types of 
questions. It is also important to consider the intent of the question. For instance, is the user’s 
intention to locate, be directed to information, or something else? This knowledge can be used to 
provide strategy tools such as search tips within a given domain. 
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Our analysis of user-intermediary interaction provided a description of the critical role that 
intermediaries play in the identification and use of information by end users. Generally, 
intermediaries share these characteristics: 

1. Knowledge of relevant publications. They keep relevant materials close at hand, and 
know what is available locally. Intermediaries identified key publications that they 
frequently used. The analysts within an agency, for example, named a common set of 
resources. These resources were kept close at hand for ready reference. Additionally, 
analysts were aware of what was available to them locally (on a Web site, a local library, 
etc.). This knowledge was used to respond to queries and to help users understand what 
the analyst could respond to rapidly. 

2. Knowledge of referrals. Uniformly, the intermediaries spoke about providing referral 
services as a critical component of their jobs. Most expressed the desire to be the last 
person the user talked to. If unable to help, they used their knowledge of other personnel 
within their organization or beyond their organization to whom they could refer the user. 
Agency intermediaries were knowledgeable about experts and researchers in their domain 
and could refer people to these experts or to their publications. 

3. An understanding of the specifics of data collection, dissemination tools, and the 
information life cycle within their domain. They understand how data are collected, 
synthesized and analyzed, how various statistics are calculated, and how they are 
disseminated and presented. They also understand the presentation formats, knowing 
which tables might be appropriate and how to interpret those tables. This knowledge 
enables intermediaries to match user queries to appropriate sources, find the most current 
sources, and provide guidance to users about appropriate use of the statistics. 

4. An understanding of how to help users express and refine their information needs. 

One of the most critical roles of the various intermediaries is to help users express their 
information needs in ways that enable the intermediary to map the need to the available 
data or refine the need so that available data can be used to address the need. 
Intermediaries conduct reference interviews in which they ask users to provide 
information about what they are seeking, how they intend to use the data, how much time 
they have available for searching and data retrieval, etc. These types of questions were 
asked by many of the intermediaries in the study. Intermediaries help users focus their 
information needs through use of probes specific to the domain. Thus in statistical 
domains, for example, users are asked about the geographic unit. Some intermediaries 
indicated that they sometimes show users some results and ask if the results are on target. 

5. Technical and searching skills. Intermediaries provide information about technical 
and/or searching skills so that users can access, print, and download data. 

Intermediaries are clearly a critical link in many users’ current processes of accessing 
information. Which of these services can be automated and which cannot? This is a critical 
question for electronic reference services, and this study points out that negotiation and 
translation aspects may be critical in getting a user to the needed information. 

Our recommendations for digital reference services based on this work are as follows: 
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1 . Consider whether it is possible to provide a digital library of key resources and experts. 

2. Provide additional guidance (either tutorials or automated tools) to improve the ability of 
users to search, download, print, and otherwise manipulate various resources. 

3. Consider whether intermediaries should provide additional interpretation services. 
Currently many analysts provide information (or pointers) but do not interpret 
information. It might also be possible to provide information on strategies that enable 
users to identify other experts. 

4. Continue to build bridges among sets of experts. 

Conclusion 

This paper has argued that valuable sources of design guidance for digital reference services are 
investigations that are grounded in the real contexts of users and their interactions with sources 
and intermediaries. Important aspects of intermediary behavior were identified through interviews 
with and observations of intermediaries in real-time -oriented settings. As digital reference 
services continue to develop, it is critical that we continue to understand and exploit what we 
know about what makes good reference service. This paper suggests some strategies and tools for 
coming to that understanding. 
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Notes 

1. For this paper, an intermediary is defined as any human that assists another in accomplishing 
information tasks; intermediaries included librarians, analysts with the agencies studied, teachers, 
etc. 

2. There have been some empirical studies of intermediary-user behavior. Ingwersen (1982), 
Nardi and O'Day (1999) and Miwa (2000) are representative examples. 

3. An explication of these interactions and the design implications may be found in Hert and 
Marchionini (1997). 

4. A value of .6 or higher is generally considered sufficient to indicate that the agreement is not 
the result of chance. 
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Understanding Intermediation in a Digital Environment: 
An Exploratory Case Study 



Silvia Barcellos 

School of Information Studies, Syracuse University 

Abstract 

The research currently being conducted investigates intermediation that occurs through an 
asynchronous text-based computer-mediated method, such as e-mail and Web forms. The goal is 
to describe and gain a further understanding of user-intermediary interactions. Its main objectives 
are (1) to investigate how and under what circumstances digital user-intermediary interactions 
occur and (2) to identify factors that might affect these interactions. The research is an 
exploratory case study of a hospital library information-provision service and adopts a naturalistic 
approach to investigate the phenomenon. This paper details the research design. Some 
preliminary results of the ongoing study will be presented at the 2nd Annual Virtual Reference 
Desk Digital Reference Conference. 



Problem Statement 

A new information-seeking scenario has been created with the pervasive use of the Internet: users 
from remote locations have direct access to information resources and human expertise. Although 
users in this situation feel empowered in their ability to access information, many may experience 
inefficiency in their search activities and uncertainty in the quality of the retrieved resource 
(Vishik, 1999). The lack of organization of information resources on the Internet, inadequate 
search engines, and difficult database interfaces (Borgman, 1996) are, among others, contributors 
to such inefficiency. Furthermore, the user cannot count on traditional mechanisms, such as peer 
review, for quality control of Internet resources. In such an electronic environment, 
intermediaries play an important role in helping users with their information-seeking activities. 

Traditionally, human intermediation occurs in a face-to-face mode where users express their 
information problems (or what they know about them) to intermediaries. The Internet brought a 
potential facilitator to user-intermediary communication as users use an electronic medium, such 
as e-mail, to interact with intermediaries. In recent years, this potential flexibility has been 
prompting information-provision services (libraries, information centers, help desks, and others) 
to offer human-mediated information services through the Internet. Despite this trend, there 
remains a lack of knowledge concerning the effectiveness of this medium. The success rate of 
these services, measured by the number of people who use them, ranges from low, as in the case 
of academic libraries’ digital 1 reference services (Schilling-Eccles & Harzbecker, 1998) to high, 
as in the case of the growing demand for AskA 2 services (Lankes, 1998). Such a discrepancy in 
information providers’ experiences calls for a better understanding of intermediation in digital 
environments. Unfortunately, researchers have given very little attention to this problem. 

Understanding the nature of user-intermediary interactions in digital environments and 
identifying factors that affect these interactions would make important theoretical and pragmatic 
contributions toward information services design and operation. The proposed exploratory case 
study aims to gain such knowledge. 




247 



Literature 



Two bodies of literature were identified as the most informative for this study: that which 
addresses interactions between users and intermediaries in information seeking processes and 
computer-mediated communication literature. Although attention has been devoted to studying 
face-to-face interactions in library settings, studies of digital interactions are scarce. Most such 
literature is anecdotal (e.g., Bristow, 1992; Roysdon & Elliot, 1988; Sloan, 1998; Still & 
Campbell, 1993; Whitaker, 1989). Few empirical studies have focused on digital reference 
services (Abels, 1996; Abels & Liebsher, 1994; Bushallow-Wilbur, De Vinney, & Whitcom, 
1996; Lankes, 1998; Schilling-Eccles & Harzbecker, 1998). Results of studies in this area seem to 
be in preliminary stages and indicate a lack of understanding of interactions occurring through 
computer-mediated communication. Such knowledge is necessary in order to implement digital 
information-provision services that better match users’ needs. 

Since this research focuses on digital user-intermediary interactions, the body of literature in 
computer-mediated communications (CMC) related to e-mail use in organizations seems relevant 
to the proposed study. Although this literature is extensive, the theories and models emerging 
from these studies are neither conclusively supported nor refuted (Fulk & Boyd, 1991; Fulk, 
Schmitz, & Steinfield, 1990; Garton & Wellman, 1995; Markus, 1994; Rudy, 1996; Steinfield, 
1992; Walther, 1992). 

Two major streams of research are prevalent in computer-mediated communications research: (1) 
that which studies how individuals choose a medium to convey a message — media choice 
theories and models, and (2) that which studies the impact of an elected medium on 
communication. Both research streams are informative to this study. The first stream provides 
insight into factors related to the medium that may affect its use for a particular task. The second 
stream of research is important because the impact of using a communication medium may also 
influence an individual’s use of it. 



The goal of this research is to describe and gain a further understanding of user-intermediary 
interactions using a text-based computer-mediated medium, such as e-mail and Web forms. 
Considering users’ and intermediaries’ perspectives, the study will be guided by the following 
questions: 

• How and under what circumstances does digital user-intermediary interaction occur? 

• What are the factors that might affect digital user-intermediary interaction? 



This research offers an exploratory case study of a digital information-provision service. It adopts 
a qualitative and naturalistic approach and employs grounded theory methods in data analysis 
(Glaser & Strauss, 1967; Strauss & Corbin, 1990). It is not based on existing theories; rather it 
aims to discover characteristics of the phenomenon through inductive analysis of data gathered in 
the field. The literature reviewed in this study is used mainly to stimulate the researcher’s 
“theoretical sensitivity” (Strauss & Corbin, 1990). 



Study Purpose 



Methodology 
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The researcher established a set of criteria for selecting a research site for the study. The major 
concern was to choose a service that does not have policies that strongly restrict user- 
intermediary interactions and that would serve as an exemplar of a digital information-provision 
service. An exemplar is not meant to be a typical site but rather one that provides opportunities 
for discovery of a wide range of factors that may be affecting the user-intermediary interactions. 
A hospital library information-provision service was chosen as the research site. Characteristics 
of the site include: 

• provides services to a variety of users, such as medical professionals, nursing students, 
patients, and laypeople 

• uses diverse media to interact with users 

• has been operating as a virtual library for two years 

• promotes a balance between accuracy, promptness and comprehensiveness in answering 
users’ requests, since the information provided by the service may serve clinical purposes 
(patient’s treatment). 

In conducting the proposed case study, the researcher will use various sources of evidence: 

• logs of text-based computer-mediated user-intermediary interactions 

• data gathered during interviews with users 

• data gathered during interviews with intermediaries 

• site observations 

• organizational publications (print or electronic), such as organizational documents, the 
information service’s brochures, forms and Web site interfaces. 

Interviews will be conducted to collect information about the contexts in which user-intermediary 
interactions have occurred. Observations of intermediaries and users will contribute additional 
contextual information to these interactions and, finally, organizational documents, brochures, 
forms and Web-site interfaces will contribute identifying organizational policies that may affect 
interactions. 

The main objective of the interviews is to elicit contextual facts and subjects’ perceptions related 
to the interaction being investigated. Interviews will be based on the log of an interaction and will 
be organized into two parts. In the first part, the researcher will approach the subject with open- 
ended questions in order to explore the context in which the interactions have occurred. (For 
example: What are the antecedents of the interactions? What are the circumstances that prompted 
the initiation of the interaction? What has happened during and after the interactions?) This part 
of the interview aims to reconstruct as much as possible the context in which the user- 
intermediary interactions have occurred. 

The second part of the interview will be semi-structured and will aim to check aspects that the 
literature or previous data analysis suggests are important in understanding user-intermediary 
interactions. 

The objectives for interviewing intermediaries are similar to those for interviewing users. 
However, interviews with intermediaries may refer to multiple interactions with various users. 
The interview with intermediaries will complement or probe what will be observed, while 
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interviews with users will be more comprehensive, as less contextual information will be known 
in advance. 

The research will be conducted in three stages: preliminary, implementation and final stages as 
explained below. 



Preliminary Stage 

This stage comprises activities that precede the fieldwork. The researcher will review the 
literature and then select and contact the information-provision service that will be used as the 
study setting. 

Implementation Stage 

This study takes a bottom-up approach, and is aligned with naturalistic and grounded theory 
methodologies for collecting and analyzing data. An iterative process characterizes the 
implementation stage, which involves emergent design, purposive sampling, data collection, and 
inductive analysis (Fig. 1). 




EMERGENT 


. Iterative . 


DATA COLLECTION 


DESIGN 


until 





redundancy 



i 



INDUCTIVE 

ANALYSIS 



Fig. 1: Implementation stage (adapted from Lincoln & Guba, 1985, p. 188). 



Data analysis is not seen as a separate stage of the research; rather it is an activity that informs 
data collection. It will be based on the constant-comparative method (Glaser & Strauss, 1967). It 
will involve coding (conceptual labeling and discovery of categories) and identification of the 
relationships among codes. Analysis will also generate a working hypothesis that will, as much as 
possible, be verified in the course of the study. 

The emergent design refers to evolving decisions during the research process of what information 
to look for next and where to gather it. The purposive sampling is a reflection of the emergent 
design and guides the following step of data collection in the iterative process. The process stops 
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when saturation is reached for those concepts and categories are considered relevant in describing 
the phenomenon. 



Final Stage 

The final stage of the research comprises reviewing and integrating findings in a case report. 
Drafts of preliminary results will be submitted to the respondents for feedback. The researcher 
will negotiate outcomes with respondents at two levels: individual and group. At the individual 
level the researcher will conduct a follow-up interview to probe unclear information, if necessary. 
Checking will be done in more advanced stages as well, when the researcher integrates the 
collected data and produces preliminary results. These results will be posted in a discussion list in 
order to receive feedback from the respondents. 

Limitations of the Study 

1. The research findings may not be fully transferable to other service environments. 

2. Data elicitation will occur, in many cases, after user-intermediary interactions are 
completed as opposed to data elicitation occurring as the interaction proceeds. This may 
restrict the validity of the information gathered because it will be based on the 
respondents’ recall of the situations. 

3. Respondents participating in the interviews are those who have had experience with 
digital interactions. So, findings related to factors that may inhibit the use of this medium 
would be restricted, since respondents who have decided not to use such a mode of 
interaction would not be participating in the study. 

Contributions of the Study 

At the research level, this study will: 

• augment our understanding of digital intermediation 

• provide models for future studies of digital intermediation 

• investigate both sides of the communication (users and intermediaries) unlike most 
studies. 

At the pragmatic level, this study will: 

• provide helpful information to improve existing information-provision services that 
operate through the Internet, since the study investigates factors that may facilitate, 
maintain or inhibit digital user-intermediary interactions 

• provide insights for training programs for intermediaries planning to work at networked 
information-provision services using a text-based medium 

• inform organizations that are planning to implement electronic reference services or other 
similar kind of services, such as help desks 

• provide insights into designing systems to manage electronic information-provision 
services. 



Notes 



'The term digital in the context of this paper implies an asynchronous text-based computer- 
mediated medium for communication, such as e-mail or a Web form. 

2 AskA services are “question and answer services that seek to fulfill the reference needs of the K- 
12 education community” (Lankes, 1998, p.9). They are digital reference services that serve a 
particular community. , 
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Becoming All Things to All People: 

Digital Reference at the Kentucky Center for School Safety Clearinghouse 

Doris D. Settles 

Kentucky Center for School Safety 

Presentation 

Introduction 

Brought into being by state legislation in 1998, the Kentucky Center for School Safety 
(CSS) has as its charge to be the central state contact point for safe school issues and 
information. We are a collaborative effort housed at four disparate (and distant) institutions; 
therefore, technology is our method of choice for communicating with each other, our audiences, 
and for providing information resources. The clearinghouse, housed at the University of 
Kentucky, functions as the information management resource for the CSS and for the state 
concerning our work. Our Web site, www.kvsafeschools.OTg , was one of the first things created 
by center staff and continues to be our centerpiece. 

Our audience is truly endless, and what we see and do runs the gamut. Questions from 
graduate students in New Zealand and school administrators in Monkey’s Paw, Kentucky are 
researched and answered simultaneously. Our Web site provides a means of communication for 
our board of directors and management team, as well as opportunities for school districts to 
become aware of grant opportunities to fund innovative projects. The home page features current 
work of the CSS as well as up-to-date articles from regional and national publications. We are 
working on Web lectures for post-secondary faculty and a Web-based course on school safety 
issues. An online mailing list, training calendar, special events calendar, and resource library 
databases provide instant access to information necessary to do our jobs or to provide 
information to our audiences. The possibilities for providing reference material are endless. . .and 
often the work seems that way as well. 

Like any multi-faceted virtual reference provider, the Kentucky Center for School Safety 
Clearinghouse feels pressured to offer the best technology has to offer. However, like any non- 
profit institution, that is frequently difficult with limited funding. Finding that balance is difficult 
at best, but we continue to strive toward that end. 



Kentucky Center for School Safety 

tiCSS 

* ** For our children, for our future 
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A collaborative effort by Eastern Kentucky 
University, University of Kentucky, Murray 
State University and Kentucky School Boards 
Association, the Kentucky Center for School 
Safety provides a dynamic blend of expertise 
in: 

♦ Project Management 

♦ Trainlngn'echnlcal Assistance to Education, Human 
Service and Justice Professionals 

♦ Law Enforcement 

♦ Teacher Preparation 

♦ Applied Research 

4 Electronic Communication 

4 School and Community Needs Assessment 
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The Center for School Safety’s mission shall be to serve as the 
central point for data analysis, dissemination of information 
about successful school safety programs, research results, and 
new programs; and in collaboration with the Kentucky 
Department of Education and others, to provide technical 
assistance for safe schools. 

In addition, HB 330 identifies nine specific areas, detailed in the 
following slides, that provide a framework for the focused work 
of the CSS. 
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Respond to requests for 
school safety information 
Research and maintain online 
resource library 
Update and maintain Web site 
at www.kysafeschools.org 
Publish KYSAFE Alert 
quarterly newsletter, E-lerts, 
Issues Briefs, and more 
Maintain state and national 
contacts in the area of school 
violence prevention 
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& $13 million to schools 

since FY 1999 
& $11 million distributed 

for FY 2001 

& Provide information on 
funding opportunities 
& Seek other funding 
sources to provide 
additional resources 






27 Annual School Safety 
Conference 

£7 Training for schools and 

communities, law enforcement 
and post-secondary audiences 
£7 Provide on-site and distance 
learning opportunities 
£3 School Safety Associates 
Program provides T/A for 
alternative programming 
27 Online consultant directory 
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d) Collaborate with schools to 
collect data which meets the 
mandates of the Safe and Drug- 
Free Schools Act, HB 330, and 
the Gun-Free Schools Act 

Independent university -based 
analysis and reporting of 
school safety data 
$ Internet-based school safety 
data collection 








¥ Identify KY-based safe 
schools programming 
¥ Compile and analyze data 
from these programs 
regarding outcomes 
¥ Online reports of 
Kentucky's Best 
Practice/Successful 
Programs 

¥ Online links for national 
Best Practices research 
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Chiefs of Police Institutes 
Justlce/Law Enforcement 
Training (JLET) School 
Resource Officer (SRO) 
Curriculum 

SRO video In production 
JLET liaison provides 
oversight and direction on 
school safety Issues and 
delivery systems 
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CD Post-Secondary Education 
School Safety Task Force 
representing Kentucky’s 26 
teacher preparation Institutions 
Q Professional development for 
university faculty addressing 
school safety Issues 
CD Web-based lectures and 
curricula for faculty use 
Q School Safety Summer Institute 
planned for summer 2001 
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Call us toll-free 

1 - 877 - 805-4277 



Or visit our web site: 
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“For every 100 men chopping at the branches of 
evil, there is one man chopping at the root.” 



Center for School 
Safety 

Eastern Kentucky University 
105 Stratton Building 
521 Lancaster Avenue 
Richmond, KY 40475*3102 
Phone: (606) 622*2898 
Fax: (606) 622-8001 

toll-free: 1-877-805-4277 
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How Many Trees in a Forest: 

Creating Digital Reference Services in Agriculture 

Melanie A. Gardner, National Agricultural Library 
Jo Ann DeVries, University of Minnesota 
Cindy Kaag, Washington State University 

Abstract 

The Agriculture Network Information Center (AgNIC) is an Internet-based, distributed system 
for quality agricultural information and resources. Currently, there are more than 35 partners, and 
each partner offers expert informational coverage of a “narrow slice” of agriculture. This 
distributed information system takes a discipline-oriented approach to bringing agricultural and 
related information and multimedia resources to the general public, the academic community, the 
business sector, and government users. Some of the goals of AgNIC include: identifying and 
evaluating major collections of electronic information and resources; facilitating access to and 
retrieval from the most useful of these resources; providing access to subject area experts and 
online reference services; leveraging the distributed character of the Internet to ensure that 
workload and responsibility are shared equally; and facilitating collaboration and communication 
among those within the broad agricultural community. Reference service is a hallmark feature of 
each AgNIC site. The authors provide a brief overview of AgNIC and two case studies. They 
describe the steps taken by the University of Minnesota in preparation to move from providing 
reference to the campus and state to the “world” and discuss efforts by Washington State 
University to build a digital reference collection. 

AgNIC In-Brief 

The Agriculture Network In formation Center (AgNIC) is an Internet-based distributed system 
for access to quality agricultural information and resources managed by an alliance of partners 
(www.agnic.org) . The U.S. Department of Agriculture’s National Agricultural Library (NAL), in 
collaboration with several land-grant institutions and others, realized about ten years ago that 
cooperation and resource sharing among institutions with common interests in agriculture are 
essential to NAL’s mission and to enhancing access to agricultural information. Shared 
resources, subject expertise, and online reference are key to the success of AgNIC. 

AgNIC grew out of a need to redefine the role of libraries and other information providers in the 
new electronic environment. AgNIC partners share a commitment to work collaboratively to 
provide access to quality resources in electronic form, including existing files, newly created 
resources, and statistical data. 

Following the idea that AgNIC ’s core vision is to provide access to quality information, online 
reference (specifically, distributed reference) became an important element of AgNIC services. 
Following a pilot among seven AgNIC subject sites (in academic year 1996-1997), partners 
agreed that the user response was reasonable and manageable. 

The network has experienced mixed success. After the pilot, and a brief period of offering online 



reference, one AgNIC subject site decided not to offer the “ask a question” option. Just recently, 
bowing to pressure from the alliance, the site added the option on their front page. In a two-day 
period, they received over 400 questions. They, again, decided to pull the option from the site. 

In a cursory analysis of the problem, there were easily explained reasons why so many questions 
were received: 

1 . The page was developed for professionals in that particular field of research. 

2. The subject is extremely “consumer-oriented,” making it a prime target for non- 
professionals and consumers. 

3. The site was developed and maintained by subject/information specialists, not librarians. 

A re-evaluation of the page structure resulted in a site redesign that decreased volume and met 
consumers’ needs. 

All but two AgNIC subject sites are maintained by librarians or have input from librarians 
regarding structure, development, and selection of resources. AgNIC partners agree that most 
users are served by sites that are well organized, built for a broad base of users, and that include 
frequently asked questions (FAQs), frequently used resources (FURs) and other general 
information. The average number of questions to a subject typically ranges from 15 to 40 per 
month. 

Many prospective AgNIC partners hesitate to join the alliance because they fear they will be 
overwhelmed by the number of questions. To complicate matters, some land-grant university 
libraries have a narrower user group than other AgNIC members. 

In keeping with the “globalization” of nearly every program, most land-grant universities are 
expanding their mission to serve the global community, and more are expressing an interest in 
joining AgNIC. 

The University of Minnesota will now discuss their efforts to make this transition in developing 
an online reference component of their subject site. 



University of Minnesota Goes Global 

The cooperative and distributive nature of the AgNIC Alliance encourages a high level of 
participation from land-grant institutions. It would be overwhelming and impossible for a land- 
grant library to expect its Web site to cover all of the agricultural and related sciences and to 
answer any question from any person. On the other hand, a Web site that specifically 
concentrates on a “narrow slice” of the subject and is supported by an established collection, 
program and research strengths, and a national reputation, is exciting and manageable. 

The University of Minnesota has two AgNIC sites. Both sites are developed around unique 
features that are the foundations of the sites. The Forestry AgNIC (forestry.lib.umn.edu/agnic) 
includes four databases that were compiled and published for more than 20 years by forestry 
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librarian, Jean Albrecht. Foresters looking for material not covered in the major agricultural 
databases use the Social Sciences in Forestry; Trail Planning, Construction and Maintenance; 
Tropical Forest Conservation and Development; and Urban Forestry databases. The Agricultural 
and Applied Economics AgNIC (agecon.lib.umn.edu/AgNIC/index.html) provides searchable, 
full-text working papers from university departments of applied economics nationwide. AgEcon 
Search is sponsored by the American Agricultural Economics Association, the USDA Economic 
Research Service, and the Farm Foundation. It is the creation of Patricia Rodkewich, agricultural 
and applied economics librarian. Economists traditionally rely on working papers to 
communicate current research. 

Online reference services are provided in the selected subject areas included in an AgNIC subject 
site. The sites include access to electronic resources available from their collections and links to 
other relevant resources available on the Internet. Many sites identify recommended frequently 
used resources and post frequently asked questions. These resources should provide the answer 
for most, if not all, questions that users will have. If there are unanswered questions, a librarian 
will personally assist a request for information. 

The differences between online reference and traditional reference desk service are not always 
obvious at first but become apparent upon reflection; online reference requires adjustments and 
experimentation on the part of a librarian. There is no opportunity to observe subtle inflections of 
the patron’s voice or body language or to ask probing questions. Therefore, the librarian alone 
determines how much or how little information is needed. Since speed is valued in this 
environment, the librarian feels the urgency to complete a response quickly, though it actually 
takes more time and effort to compose a written response than it does to reply verbally. Perhaps 
the most striking difference is the anonymous nature of the requests. Unless the site’s form 
collects user information, the librarian has no context for understanding the request. Feedback 
about the usefulness of the answer is minimal online as compared with traditional reference 
interactions (walk-in and telephone). 

A properly designed, well-organized structure of carefully selected resources will ensure that 
users can easily locate needed information and limit the number of requests for additional 
information. Web site development and online reference are time-consuming and high 
maintenance services. Reference workflow has shifted dramatically from blocks of time at the 
reference desk to development of Web-based reference. Web users expect service twenty- four 
hours a day, seven days a week. Interestingly, many queries are in the form of comments 
praising the usefulness of the databases or suggesting new links or recommending improvements 
in format. An important factor to consider when conceptualizing a Web site is the anticipation of 
the broadest possible scope of users. Creating layers of resources is a way to accommodate the 
breadth of user requests. 

Anticipation of the broadest scope of users is a new direction for most land-grant libraries. 
However, with a re-emphasis on the outreach mission at the University of Minnesota, the broad 
scope of users complements institutional goals. The university actively initiates partnerships with 
business, industry, undergraduate education, rural communities, K-12 education, and 
international research programs. The university faculty and students work and live worldwide. 



Collection Development for Virtual Reference 

Washington State University (WSU) concentrates on tree fruit as its primary topic of agriculture. 
Washington State is a major producer of tree fruit, providing 47% of America’s total annual 
output of apples and cherries and 42% of its pear output. WSU has several extension sites 
dealing with tree fruit and produces numerous publications on a yearly basis. Building on this 
position of strength, WSU undertook to develop the AgNIC site for tree fruit. 

If the subject coverage of an individual AgNIC site is narrow, the intended audience is certainly 
broad, comprised of the general public, the academic community, businesses, and government 
agencies. There are substantial implications for developing a reference collection to serve many 
different groups. We could legitimately have links ranging from WSU’s own popular “Ask Dr. 
Universe” science site for kids to extension publications to economic indicators to federal 
regulations governing the number of port-a-potties provided to migrant field workers. The 
following section discusses our efforts to build an electronic reference collection on tree fruit to 
serve a diverse clientele. 



Goals 

Three of AgNIC’s goals are particularly pertinent to providing reference information. We aim to: 

• Identify and evaluate major collections of electronic information and resources, 

• Facilitate access to those collections, 

• Provide access to experts in the field and reference services online. 

Collection Development for the Site 

Traditional collection development is the basis for online collection development. The tools have 
changed, but the concepts remain the same. Librarians add value to resources by locating, 
evaluating, and providing access. The process is incremental: 

• reference 

• reference sources 

• electronic reference sources 

• online electronic reference sources 

Several library publications now provide reviews of sites, both free and for a fee. Examples are 
“Internet Reviews” in College and Research Libraries News, “Databases” in Reference and User 
Services Quarterly, and Choice, which integrates print and electronic media. Subject-specific 
journals have been slower to review electronic resources; we have yet to find any journals that 
regularly review Web sites. Consequently, our librarians search for and evaluate sites. 

So, what is different about selecting electronic tools for virtual reference users, or for in-house 
use in answering virtual reference questions? An obvious issue is access: can the user access a 
particular tool? Is licensing involved, or are there hardware or software restrictions? The selector 
must look at these issues in addition to the usual evaluation of authority, timeliness, reliability, 
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usability, depth, breadth, etc. Is the Web site easily navigated? Are the results accurate and 
consistent? Can the user easily exit the site? Institutional policies can guide decisions. 

Searching for virtual resources requires thinking globally. Librarians cannot rely on a few library 
review sources or a book approval vendor. There is no substitute for actively searching for 
resources, which means spending time online trying different strategies in different search 
engines. The same search in Google, Ask Jeeves and AltaVista will result in very different 
results. For example, I recently ran a search for “Yellowstone science” and checked a dozen 
engines for relevancy on scientific issues in the Greater Yellowstone Ecosystem. Total retrieval 
and actual usefulness of the first 100 hits from each overlap by no more than 20%. In providing 
virtual reference services and collections, librarians sort the good from the bad. My alma mater, 
the other UW in Wisconsin, embraces the motto, “fearless sifting and winnowing”; that’s what 
collection development is, especially in the face of the flood of possible electronic resources. 
Select the few from the many and make those accessible. 

Acquiring 

This is where you need to be a cross between “Miss Manners,” an accountant, and a lawyer. 

Once you have located resources you want to add to your own virtual reference site, you have to 
be aware that not everything is free for the linking. You have an obligation to make sure that you 
are not abusing the Internet. Unless a site is identified as freely accessible, professional courtesy 
requires you contact the provider to request permission to build a link into your site. I recently 
received a flyer for a wonderful resource produced by our neighbor institution, the University of 
Idaho, in print and CD-ROM. Since WSU comprises several campuses, research stations, and 
extended degree locations, I called to check on licensing for mounting the CD-ROM on our 
server so that all our patrons could access it. I was told we could simply catalog their Web site 
into our online catalog, so users would always have access to the latest information and updates 
at no charge. Of course, this is a best-case scenario. 

Free sites are likely to be provided by academic institutions, government agencies, and 
businesses selling tangible products. Businesses selling data - bibliographic, economic, and 
statistical - will not appreciate libraries providing links to their sites from a single subscription to 
the world in general. You will need to negotiate licensing if you want to add such sites to your 
virtual reference collection, whether you are dealing with something as far-flung as an AgNIC 
site or with your own local reference collection. Can you afford a license which will provide 
access to distributed user? Or will you have to make do with a single-station license that you can 
use in answering reference questions? Licensing and lawyering have become a significant part of 
contemporary collection development and require a great deal of attention. 

Cataloging 

Organizing a virtual reference collection into sub-topics is in a sense cataloging in and of itself. 

If you are going to provide a search engine for your site, you will need to be sure you allow 
searching on author, publisher, subject, and keyword information, as well as the title. 

Real problems occur when librarians list electronic resources, onsite and off, in the library 
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catalog itself. All librarians are in some sense reference librarians: we’re good at finding things, 
and we help others find things. We generally want that “finding” to be an easy process, and 
therefore we not only select and acquire, we catalog as well. When we buy access to an online 
index or to a CD-ROM or an e-joumal, we expect that resource to be reflected in our catalogs. 
But what if the resource is a free site? Is that added to the catalog? Who will check to see if the 
link keeps working, and fix it if it breaks? What if a former print product is now only available 
electronically? Do you link to the old record, wait for a new one from LC, create your own new 
one, or put the resource on a separate list of electronic resources? If you are going to have an 
effective virtual reference collection, it is important to discuss these issues and have a policy in 
place, or you’ll be swamped with individual item questions and your catalog will be neither 
consistent nor helpful. 



Collection Development for the Service 

Reference service is so basic to our profession that we sometimes forget how it has changed and 
how it varies from sector to sector. In academia, we teach classes on how to use reference 
resources, and we are likely to take individuals on a talking tour of where we look to answer 
their questions. We are likely to provide users with a call number or document delivery 
information to pursue on their own. In a business library, the librarian usually finds the 
information requested, obtains any documents, and often creates a precis to go along with the 
packet. To support both extremes of service, the reference collection will evolve to fill the needs 
of the users. But what about the reference collection supporting virtual reference services? 

Two main areas need to be considered: resources for the librarian answering virtual reference 
questions, and resources for the virtual user. Recently I received an e-mail reference question 
asking for information on three medical syndromes, two of which were misspelled. If this had 
been an in-person question, I would have taken the user to our medical reference area, showed 
the user how some appropriate dictionaries and encyclopedias worked, gently showing entries 
under the correct spellings. For a virtual user, I could not do that, nor would I offer medical 
advice to any user. I answered the question with short descriptions of each syndrome from a print 
medical dictionary, suggested checking the local library collection, and then did a Web search 
for reliable sites. I found good, Mayo-Clinic level information readily and included the URLs in 
my response. 



Conclusion 

Being in an information profession in the “age of information” is an exciting, stimulating, and 
terrifying situation. We are redefining reference services and collections daily. Virtual reality is 
rapidly becoming just plain reality and virtual reference collections are a basic tool of any 
reference service. Collection development practices have to evolve to take into account both the 
reference site and the reference service - and so do reference librarians. 
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The Electronic Information Desk: Communication Made Virtual 



Jessica Albano, Adam Hall, and Lorena O’English 
University of Washington Libraries 

htt p://www.lib.vvashimJton.edu/siizrcf/LibOiiest/ 

Electronic information desk services are exciting opportunities to connect library patrons who 
have questions about library services, policies, and other information with the answers they need. 
When the University of Washington Libraries switched to a "home-grown" online catalog in 
1992, the innovative alliance between the library and the campus Computing and 
Communications (C&C) office that created the catalog system, Willow, also created a text-based 
comments service that allowed patrons to ask questions about the new catalog. 

As Willow expanded to include locally-mounted databases, the number of questions increased. In 
the beginning, questions were mainly of a technical nature, and were answered equally by a 
library technician from the Libraries Electronic Information Program and staff from C&C. 
Gradually this began to change, as users became familiar with the catalog and database interface 
and asked questions about search strategy and their search results. Over time, questions were 
increasingly being referred to other library units and staff members, and LibQuest, an electronic 
information desk, was created (http://www.lib.washington.edu/suzref/LibQuest/). LibQuest staff 
implemented an extensive awareness campaign within the Libraries to make sure that all library 
units were aware of the service, that inquiries were referred to the correct library unit, and that 
they were responded to quickly and effectively. 

Over the years LibQuest has evolved into a one-stop-shopping information and referral service 
for library patrons. The migration of the library catalog from Willow to a Web-based Innovative 
Interfaces catalog, the major expansion of the number of available Web-based library databases, 
and the implementation of the Libraries Information Gateway have been significant factors in this 
evolution. Patrons e-mail LibQuest directly or through “Contact Us” links on every page of the 
Gateway. The "LibQuest Decision Tree" diagram shows the process used to refer, respond, and 
archive LibQuest inquiries. 

E-mailed questions are answered by LibQuest personnel or referred to the appropriate person 
through a network of participating library staff members. A system of tracking questions and 
responses ensures that all questions are answered as quickly and completely as possible. Over 
time, the involvement of C&C has almost totally disappeared. LibQuest is staffed by two 
librarians. This has meant that short-answer and general reference questions can now often be 
answered directly, without having to be referred to RefQuest, the Libraries e-mail reference 
service for UW-affiliated patrons. 

A recent analysis of LibQuest inquiries over a two-month period shows that while two-thirds of 
LibQuest users are affiliated with the UW, many questions are received from people all over the 
United States, and indeed the world. The pie chart “LibQuest Users” shows a breakdown of 
inquiries by source over two months. The questions asked cover many topics as well. Most 
inquiries are referred, including many questions about circulation and library fines or questions 
that should be answered by a specific library unit. Increasingly questions have to do with remote 
access to UW-restricted databases; the implementation of a Proxy Server allowing people who do 
not connect to the library system through the official UW computer service to get access has been 




very popular, but also brings many technical questions. The pie chart “Questions by Category” 
shows a breakdown of the kinds of questions received over a two-month period. 

In the future, LibQuest intends to fully integrate interactive Web technology to create an even 
more virtual information desk. Internet “chat” services, that allow real-time communication 
between library staff and patrons, and elaborate Web-based services, that would allow library 
staff to work with users directly while both are seeing the same screen (similar to those being 
used by such e-commerce vendors as Lands End), are being considered. 

LibQuest is an integral part of the UW library system now; this service bridges the gap between a 
large 21 -library system with a multitude of units, divisions, and subject specialists, and the 
specific needs of a single user. 
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Enhancing Digital Reference at the Georgia Institute of Technology 

Bruce Henson 

Georgia Institute of Technology 

Presentation 

Introduction 

Since October 1994, the reference department at the Georgia Institute of 
Technology’s Library & Information Center has provided the ASK A Librarian virtual 
reference service to students, faculty, and staff. In the past year, three new components 
have been added to enhance ASK A Librarian: a client form, an ongoing assessment 
survey, and a FAQ database. The department also introduced Real Time Reference, a 
new digital reference service that uses “chat” software and enables the electronic 
transaction to occur synchronously. 

The ASK A Librarian client form was implemented in Spring 2000 and replaces a 
previous client question box. The form’s purpose is to more closely approximate the 
information gathered from a traditional in-person or telephone reference interview. The 
standardized form collects pertinent information from clients, decreases the number of 
multiple communications between client and staff, and reduces incorrect assumptions 
made by staff. The FAQ database will be implemented in Fall 2000 and will allow 
keyword searching of three question fields: e-mail subject line, the question, and the 
response. All questions in the database are stripped of any identifying information about 
the client. An ongoing assessment survey was implemented in spring 1999. ASK clients 
are e-mailed at the end of the month in which they ask a question, with their original 
question included in the e-mail. They are asked to complete an anonymous Web-based 
survey consisting of both open-ended and closed questions. 

Real Time Reference was implemented in May 1999 and uses America Online 
Instant Messenger “chat” software to enable clients to ask and receive answers to 
questions while on-line. Statistics show that usage of the service is growing. The service 
has been advertised in brochures sent out to all dorms, the library home page, and the 
closing signature in all ASK A Librarian replies. 
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Question Using Client Form 

The client question asks how to search for journal articles, which would normally occasion a standard 
response about how to search for articles. The question on the form concerning previous research, 
however, shows that the client had searched for journal articles in the catalog, which provides valuable 
information lo staff. In our reply the difference between catalogs and article databases can 
explained, 

Name: xxxx 
Email: xxxx 
Phone: xxxx 
User Type: Staff 
GT Access: Yes 

Reference Question: How do you look up journal articles? 

Previous Research: 1 have tried using GTEC but all of the titles and 
authors come up - 'no matches' 

Kinds of Materia Is: 10 journal articles dating as far back as 1959 to 1993 
Search Keywords: name ofjoumals 
Deadline: need by 5/12/00 
Other Requirements: 
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E-mall to ASK Clients Requesting their Participation In the Survey 

Ask A Librarian Anonymous User Survey 

Thank you for recently using the Georgia Tech Library ASK A Librarian Digital 
Reference Service. In order to continue improving service to clients, wc arc conducting a 
brief survey. The Georgia Tech Librajy respects the privacy of its users. Participation in 
the survey is voluntary, and all information will be kept confidential. Your original 
question and the reply from Ask a Librarian is included below. 

Please take a few minutes to answcT the following questions. 
l)tip:/'Avww. libra rv.nalechxdu' , asksurvcv frame, httn 

Your help is greatly appreciated. Thank you for your time and cooperation. 

Sincerely, 

Bruce Henson 

Assistant Head, Reference 

Digital Reference Coordinator 

Georgia Tech Library & Information Center 

(404)894-1390 

brucchcn^oii'i-'librtiiv.v’atccli.cdu 
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Survey Question # 2 • Have you used ASK A 
Librarian more than once? 




Survey Question # 3 - Did you receive a response 
to your question within 24 hours? 





































□ < 24 Hours 


— 






□ > 24 Hours 


























] 







ep-wo ism f*r. iica 0^.3 



Survey Question # 4- Was the response to your question 
presented In a clear and understandable way? 
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Survey Question # 6 - How would you rate the 
usefulness and effectiveness of ASK A Librarian on 
a scale of 1-6? 
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Collaborative Digital Reference Service: 

Library Quality Reference Service Meets the Web 

Diane Kresh 
Library of Congress 

Presentation 

Introduction 

Kresh presents the most recent efforts of the Library of Congress to develop the 
Collaborative Digital Reference Service (CDRS), an initiative consisting of an 
international network of national, academic, public, and special libraries. Results of pilot 
phases and future plans for CDRS are discussed. 
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Net Librarian: A Danish National Online Information Service 



Vera Daugaard 

Heming County Library, Denmark 
Abstract 

Net Librarian is an Internet-based information service run in cooperation by three Danish 
public libraries and supported by the Danish Library Authority. The service has existed as a 
pilot project since October 1999 and is expected to develop into a permanent service in 2001. 
Net Librarian’s Web page (www.biblioteksvagten.dk) includes an inquiry form that enables 
citizens from all over Denmark to ask questions at any time. This paper discusses the 
development of Net Librarian and describes challenges encountered such as limited financial 
resources, marketing, and consistency of service among three separate organizations. The 
future of Net Librarian is also addressed. 

Introduction 

Danish public libraries are financed through public funds and the staff is salaried, professional 
or not. The average expenditure on public libraries is nearly $50 per inhabitant per year. 
Voluntary programs are non-existent. Due to financial cuts in recent years, it has become 
clear to many Danish public libraries that cooperation between libraries is necessary in order 
to develop modem library services. 

An example of this type of cooperation is Net Librarian, an Internet-based information 
service, where citizens may submit questions at any time through the Web and receive an 
answer within 24 hours. Net Librarian also offers a phone service in the evening, when many 
Danish libraries are closed. Net Librarian is run in cooperation with three Danish public 
libraries in Gentofte, Heming and Silkeborg. The project started in 1999 and was originally 
scheduled to run until September 1, 2000. It is now expected to develop into a permanent 
service. Besides examining whether there is a need for this kind of national information 
service, the aims of the project are to: 

• Identify organizational and technical problems resulting when three public libraries 
with different organizational cultures develop common work 'routines and methods for 
servicing the end user. 

• Examine the extent to which inquiries can be answered independently of the physical 
library, using information available on the Internet only. 

• Assess users’ needs in terms of hours of use, types of questions, and geographic 
location and age of users. 

• Determine the professional demands on the librarians staffing the service. 




Web Site 

The Net Librarian Web site is found at www.biblioteksvagten.dk (the site is currently in 
Danish only). Since Net Librarian is aimed at users of public libraries, it was agreed that the 
Web site should be simple and easy to use. The Web site does not contain any fancy graphics 
that might slow down access to the page. The main component of the Web site is a two-tiered 
inquiry form. Net Librarian attempted to keep the inquiry form short and discouraged 
personal questions. The form allows staff to gather sufficient information to compose a 
quality answer. The upper half contains fields for the user’s name, e-mail address, postal code 



287 



i 



and town, and date of birth, and a text box for the question. This part also includes a dispatch 
button. It is optional for users to fill in the lower part of the form, which contains information 
about the purpose of the question, the user’s job/education, phone and fax numbers, and 
address. This information helps staff to locate the correct answer to the user’s questions and 
also allows the staff to send material by mail. 

The Web site also contains the following features and information: 

• Links to Danish subject-specific information services on the Internet (e.g. Ask Science, 
Animal Doctor and Denmark’s Family Lawyer) - It is unclear whether many of the 
visitors to our Web site choose to ask their questions at these other services, but we do 
sometimes refer questions to the services ourselves. 

• A link to the Public Libraries’ netguide (www.fng.dk) - This netguide consists of about 
2,600 links - mostly in the Danish language - to quality information resources on the 
Internet. All links are chosen by Danish librarians, supplied with annotations detailing 
content, publisher, etc. and classified into 21 subject groups. Each librarian is responsible 
for updating the links in a specific subject area. 

• Factual information about Net Librarian - Hours of operation, phone and fax numbers, e- 
mail address, response time, etc. We also inform our potential users about service 
response policies: Net Librarian points users to information but does not interpret the 
information, carry out exhaustive literature searches, or provide legal guidance of any 
kind. 

• A page with “bouncing answers” - answers that could not be sent to users because 
incorrect addresses were given. 

Net Librarian is currently adding a chat function to the site for the purpose of real-time 
interaction between Net Librarian and the user. However, the service has not yet selected an 
appropriate software product. Products under consideration included NetMeeting (which has 
the obvious advantage of allowing us to share documents with the user, but couldn’t function 
through the firewalls of our three libraries), ICQ, and Webline (which was too expensive). 
Our investigations continue. 

An “administrative” Web page has been constructed for participating librarians containing: 

• Questions sent to Net Librarian via the inquiry form, searchable by answering 
librarian, title words, question, and answer (questions not yet answered can be sorted 
out) 

• Inquiry form for questions received by phone, fax, or e-mail 

• Link to common mail program 

• Names, addresses and phone numbers of the participating librarians 

• Duty roster 

• Instructions (for switching phone and fax and using the common telephone answering 
device) 

• Net Librarian statistics 




Staffing and Administration 

Net Librarian is supported by the Danish National Library Authority and its financial 
resources are the responsibility of Heming County Library. The service is formally managed 
by a steering committee, comprised of one leader from each library, the project manager, and 
the project researcher. The steering committee sanctions all decisions concerning financial 
matters, opening hours, marketing initiatives, etc. 
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The daily work of Net Librarian is planned in the project group, consisting of two librarians 
from each library, the project manager, and the project researcher. The project group is 
responsible for the content of the Web site and its construction and all discussions of 
professional matters. 



Building Cooperative Service 

One of the purposes of Net Librarian is to identify organizational problems resulting from the 
cooperation of three public libraries in providing service to the user. A start-up seminar was 
offered for the participating librarians in order to establish a common professional basis for 
the future work, and also to encourage social relations between the librarians located in 
different geographical areas. 

At the two-day seminar, “Common Beginning - Common Learning,” each library gave a 
short presentation highlighting their collections, services, cultural issues and local 
communities. Although Denmark is a small country, these presentations showed striking 
cultural, social and political differences between locales. The three libraries, however, also 
have much in common: they participate in many pilot projects and they want to play a virtual 
part in the future development of Danish public libraries. 

The seminar created a sound basis for cooperation within Net Librarian. The project group 
currently discusses problems and differences, which are bound to occur when the service 
expands to include more libraries. Ideally, each library should be able to benefit from each 
other’s answers and sometimes finish an answer started by one of the other net librarians. 

The cooperative aspect of the service is transparent to the user. The user does not observe a 
significant difference whether the answer is given by a librarian from Gentofte, Heming or 
Silkeborg. Net Librarian assumes a common responsibility, based on trust that colleagues in 
other libraries can deliver an answer of the same quality. 

One issue discussed within the project group is how to answer student questions. Some 
members of the project group are very aware that Net Librarian must never make it 
unnecessary for students to use their local library. Searching and locating information is 
generally a virtual part of a student’s assignment, and using Net Librarian tends to be 
regarded as “cheating.” In recent years, all Danish public libraries have been working on 
making students more and more resourceful: we point them to the computers and give them 
some guidance, but we expect them to carry out the search themselves. To some extent, Net 
Librarian finds the answer. An inquiry to Net Librarian should lead the user one step further. 

I am confident that Danish librarians will soon find a way to make students resourceful on the 
Internet, too, simply by making Internet-based user guidance available. We will have to 
accept the fact that the use of Net Librarian and other Internet-based library services replaces 
visiting the physical library to some extent. 

Marketing of Net Librarian 

Although we consider Net Librarian a quality service, we know that only very few 
people have ever heard of it. Our financial resources are limited and resources for marketing 
are insufficient at about $2,500. The three libraries have put some of their own financial 
resources into the project, but marketing remains a weak point. To many Danish public 
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libraries, Net Librarian is considered a competitor, because it is available to all Danish 
citizens and answers questions that would otherwise be addressed by the user’s local library. 

To minimize the competitive aspect, Net Librarian sent a letter to all the main libraries of the 
Danish local authorities at the start of the project. The letter explained the ideas behind the 
project and encouraged the libraries to add a link to Net Librarian from their own Web sites. 
About one third of the libraries have chosen to do so. 

Marketing efforts have included: 

• A press release sent to all the national media - Net Librarian was actually mentioned in 
almost all national newspapers. A few of the newspapers included articles on Net 
Librarian. 

• Registration with a number of Danish search engines - We succeeded in getting the 
service searchable on the Web sites of various Internet guides in the world of 
education. 

• Local marketing — Interviews were conducted on local radio stations and in local 
papers in the three communities. 

• Bookmarks - A Denmark-based firm named “Bog Card” produces and publishes 
bookmarks according to customers’ requests and distributes them to libraries and 
bookshops throughout the country. The customer pays for a number of bookmarks and 
for a certain period of display. At the end of this period, the libraries and bookstores 
return the remaining bookmarks to “Bog Card” and the customer receives an account 
of the marketing success. Though this service is rather expensive, the project group 
recommended “Bog Card” to the steering committee, which approved the action. From 
March 14 to April 4, 60,000 “Bog Cards” were displayed in 21 1 libraries and 79 
bookshops, and the effect was noticeable; Net Librarian was busy during that period. 

• Press material to all Danish community papers - A poster and folder will be sent to 
selected groups of educational organizations and libraries all over the country. 

• Journal Articles - Articles have been written in a number of Danish library journals. 

The Future of Net Librarian 

Net Librarian’s goal has been to develop into a permanent, nationwide service in the Danish 
library system. The future of the service is still uncertain, but the project’s steering committee 
has applied for financial support from the Danish National Library Authority’s Development 
Pool for Public and School Libraries. If money is awarded, it will be spent on: 

• Improving the service, based on researchers’ recommendations 

• Involving more libraries in the project 

• Developing a permanent model for organization, management and financing 

The pilot has demonstrated a need to expand hours of operation. Net Librarian is now 
available during limited hours from October through March only: These hours were chosen 
based on the project group’s assumptions about users’ needs. 

In the future, Net Librarian will be part of a joint Web portal for all Danish research libraries 
and eventually public libraries. This Web portal is scheduled to be developed within two 
years. When the portal is created, Net Librarian will be a significant component. 
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Ask A Question! 

A Collaborative Virtual Reference Service 

Valerie Footz 

Northern Alberta Institute of Technology 

Presentation 

Introduction 

Ask A Question is a project of libraries in post-secondary institutions in Alberta, 
Canada. This presentation traces the background, development and structure of the 
service. It also discusses issues raised over the course of the project. 

In the fall of 1998, a group of librarians at three post-secondary institutions in Alberta 
- Grant MacEwan College, Northern Alberta Institute of Technology (NAIT), and Red 
Deer College - agreed to collaborate on the development of a shared electronic reference 
desk. After securing Alberta Government Knowledge Network funds, a pilot project was 
begun. The project had four goals: 

• Assist users in effectively accessing and utilizing a wide range of electronic 
information sources; 

• Create a repository of information from all participating institutions for users to 
draw upon when reference assistance is not available due to time or place; 

• Provide data on users and their approaches to information search and retrieval; 

• Determine if shared responsibility for the provision of virtual reference service is 
feasible. 

A review of current library services and existing virtual reference services was 
undertaken. No satisfactory software application was found to meet the requirements of 
the project, so the team decided to develop an in-house solution. The result was Ask A 
Question, which was launched on March 6, 2000. This innovative service has many 
unique features. Besides allowing students to submit questions through the Web, it offers 
a continuously updated, searchable database of questions and answers. 

The question-answering process is as follows: 

1. The user clicks on the Ask A Question icon on the participating institution's 
library Web page. 

2. The user then logs in (after an original registration process) and is taken to the 
Ask A Question submission form. 

3. The submitted question is stored in a database. 

4. Ask A Question staff at all three institutions are alerted to new questions. 

5. Answers are found and are added to the database, which automatically sends 
the answer to the user's email address. 

The strength of Ask A Question is its custom-built back end. Ask A Question 
employs open systems technologies, including PERL and SQL. Staff members have the 
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option of transferring ownership of questions or sending them back to the new list. The 
integrated e-mail/database workflow allows for the easy management of questions and 
answers. A collaborative relationship among library staff across institutions is fostered as 
a result. 
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Outline 



Ask A Question! 



A Collaborative, Virtual 
Reference Service 




The Facets of Digital Reference: 

VRD 2 nd Annual Digital Reference Conference 
Seattle, October 16-17, 2000 



• Background 

• Development 

• Structure 

• Concerns 

• Future 





Purposes of the Project 




^To assist users in accessing and 
utilizing a wide range of electronic 
information sources effectively 



XTo create a repository of information 
that users can draw upon when 
reference assistance is not available 
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Purposes of the Project ^$jf 




Project Chronology ^!jf 


l|To provide data on users and their 
approaches to information searching 
and retrieval 




• Spring 1998 - Original concept 

• Fall 1999 - Autonomous Ask A Question 
services launched 

• Nov 1999 - Stakeholder meeting held to 
examine technology options for collaboration 

• Dec 1999 to Feb 2000 - Beta development 




fSTo determine if shared responsibility 
for the provision of virtual reference 




• Mar 6, 2000 - Ask A Question collaborative 
service launched 

• May 2000 - Emerging Projects Grant received 


service is feasible 




• August 2000 - 3 more libraries added 

• Undated chronoloqv 








Suggestions 


iff 


• Know your library and its context 




• Know the objective of the service 




• Know your users 




• Keep current with technology 




• Learn from other initiatives 




• Listen 




• Compromise 




• Revise as necessary 
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Design Considerations and <«* 




Design Considerations and 


Objectives ^|§ 




Objectives (Cont.) 


• Access from anywhere 




• Data ownership model 


• Rapid development 




• Cross-institutional collaboration 


• No funding for hardware/software 




• Highly automated workflow 


• Open systems 

• E-commerce principles / Tush" 
architecture 

• Scalability, sustainability 




• Searchable database of answers 



Technical Implementation 

• Requires Internet Explorer 4, Netscape 
4.X or compliant with JAVA and cookies 
enabled 

• Statistics extracted to another program to 
compile and analyze 

• Authentication is based on e-mail 
addresses only 




Participating Libraries 

• Some policies and procedures must be mutually 
agreed upon: 

• Turnaround times 

• Application of subject categories 

• Permissions for administrative tasks 

• On what basis questions will be chosen for cross 
institutional ownership - date, subject, shift 

• But some decisions must be made at the 
participating library level: 

• Who participates 

• On basis of ownership within own library 
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Features 



Components 



User 

profile 


Question 
and answer 


Staff 

module 


Survey 




• Captures the reference transaction in a 
searchable database 

° Concept of ownership of questions and 
individual staff responsibility for answering 
those questions 

° Ease of submitting questions and answers 

* Search results are “bookmarkable” 

° Survey distributed with answer 








Benefits to Users Include: 


ill 


* 24/7 access 




• Level of anonymity 




* Answers sent directly to e-mail 




• Customized responses 




• Quick turnaround 




• Pull from expertise from many institutions 


* Dynamic searchable database of 




previously asked Q & A’s 
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Roles 




Concept of Ownership 


• Administrator 




• Staff take “ownership” of questions 


* Liaises with IT staff, coordinators, and advisors 




• Means a commitment to answer it within 


• Handles problems and questions 
• Coordinator 




the agreed upon guidelines 


• Acts as representative of participating institution 




• Ownership may be transferred between 


• Liaises with administrator and other coordinators 

• Acts as backup for staff at institution 
• Staff 

• Answers questions 




staff or institutions if necessary 
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Benefits to Staff include: 

• 24/7 access 

• Asynchronous written transaction 

• Level of anonymity 

• Pull from expertise from many institutions 

• Dynamic searchable database of 
previously asked Q & A’s 

• Easy cohesive Q & A management 

• Provides user and collection information 





Staff Question Management 

• Send back to new list 

• Transfer ownership 

• Refer questions 

• Pause questions 

• Edit questions (administrator only) 

• Delete questions 




Concerns 

• Spamming 

• Wrong e-mail addresses 

• Authentication 

• Knowledge of other institutions and their 
resources 

• Confidentiality 

• Retention of data 

• Policies and procedures 
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For more information, contact: 




Valerie Footz 

Instructional Librarian 

Northern Alberta Institute of Technology 




Edmonton, Alberta, Canada 




valerief(a)nait.ab.ca 




or 




virtual® veats.qmcc.ab.ca 









Future 


lit 


• Better statistics 




* User information management 




* Integration with other systems 




* Scalability 




* Sustainability 




* Quicker turnaround guarantee 




* Promotion 
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Building Energy Science and Technology Digital Collections 
for an Information Infrastructure for the Physical Sciences 

Karen J. Spence 
Mary V. Schom 

U. S. Department of Energy, Office of Scientific and Technical Information 

Abstract 

The U.S. Department of Energy Office of Scientific and Technical Information (OSTI) provides 
a suite of innovative digital information resources. Included in these resources are world-class 
products that address the three main ways by which researchers disseminate their findings: the 
DOE Information Bridge (grey literature), PubSCIENCE (journal literature), and the PrePRINT 
Network (preprints). These products are key components of the suite of resources provided 
through EnergyFiles, a virtual library of energy-related scientific and technical information. Each 
product can be searched individually or in parallel with other energy-related resources using 
EnergyPortal, which is the groundbreaking distributed search mechanism of EnergyFiles. This 
history of success lays the foundation for OSTI’s new initiative, a future Information 
Infrastructure for the Physical Sciences. 



Introduction 

The Department of Energy (DOE) is among the leading research agencies in the world, investing 
seven billion dollars annually in research and development (R&D). It is vitally important for 
research agencies to disseminate their information as broadly and as quickly as possible, 
providing access to data and information that fuels essential knowledge. For over 50 years, 
DOE’s Office of Scientific and Technical Information has been collecting, preserving, and 
disseminating the department’s scientific and technical information (STI). By utilizing 
information age technologies, OSTI has radically changed its information services and has 
developed a group of award-winning Internet resources that bring science information to the 
desktop at no cost to the end user. These resources provide easier, faster, cheaper, more 
complete, and more convenient means of accessing and using global STI by scientists, 
researchers, academia, industry, and the public. 

One-stop shopping is provided with EnergyFiles, a Web-based virtual library of both DOE and 
worldwide energy-related scientific and technical information. Its search mechanism, 
EnergyPortal, is easy to use and integrates parallel searching across heterogeneous and 
geographically dispersed databases and Web sites. Key components of EnergyFiles are DOE 
R&D Project Summaries (current research), DOE R&D Accomplishments (outcomes of past 
research), the DOE Information Bridge (grey literature), PubSCIENCE (peer-reviewed journal 
literature), and the PrePRINT Network (preprints). 

DOE Information Bridge 

The DOE Information Bridge ( www.osti.gov/bridge ), available since April 1998 in collaboration 
with the U.S. Government Printing Office (GPO), contains DOE research and development 
reports in physics, chemistry, materials, biology, environmental sciences, energy technologies. 



engineering, computer and information science, renewable energy, and other topics. It includes 
more than 55,000 full-text reports from 1995 to the present, in over 4.3 million pages. It provides 
free, convenient, and quick access to full-text DOE research and development reports. Users 
remotely access and download the reports free of charge and in significant volume. 

The DOE Information Bridge focuses on providing access to scientific and technical reports 
produced by DOE, DOE national laboratories and DOE contractors. New reports processed by 
OSTI are added routinely and legacy reports are added as resources permit. Since its 
introduction, the content of DOE Information Bridge has more than doubled, and it has been 
most favorably received. Among the awards and accolades received are: 

• Commendation by the Depository Library Council 

• Vice President Gore’s National Performance Review Hammer Award 

• DOE Information Management Technical Excellence Award 

• Inclusion in the October 1, 1998 inaugural issue of Access America Online Magazine, a 
product of the Government Information Technology Services Board 

• Favorable review in the University of Wisconsin’s “Scout Report” for science and 
engineering 

• Citation by the Global SchoolNet Foundation and Yahoo (Pick of the Day and Week) 

The basic search option offers specific data fields and the advanced option incorporates Boolean 
operators to increase search precision. Users can search the entire collection (full-text and 
bibliographic data) or portions of it. Full-text page images may be viewed in GIF, PDF, or TIFF 
formats. PDF (image only) and the original input format are available for downloading full-text 
documents. 

Building and expanding the DOE Information Bridge reinforces DOE’s and GPO’s commitment 
to make DOE research reports available and to move federal programs and activities into the 
ever-expanding world of the information age. 

PubSCIENCE 

PubSCEENCE ( www.osti.gov/pubscience ) was developed to facilitate searching and accessing 
peer-reviewed journal literature in the physical sciences and other disciplines of interest to DOE. 
Made available in collaboration with the Government Printing Office (GPO) in October 1999, it 
provides for quick, easy, and free searching of 1.8 million citations and abstracts from more than 
1,000 journals. Hyperlinks provide access to full-text articles if the user or organization has a 
subscription to the journal. If the user lacks such a subscription, access to the full text can be 
obtained by pay per view, by special arrangement with the publisher, by library access, or 
through commercial providers. 

Not only is the Internet changing the way publishers think about publishing, but it has also 
impacted government’s dissemination of scientific and technical information. PubSCIENCE is 
an outstanding example of converging interests: the user’s desire to access current scientific and 
technical literature, the department’s desire to facilitate the flow of peer-reviewed scientific and 
technical information, and publishers’ interests in obtaining the widest possible visibility for 
their published materials. 

erJc 




PrePRINT Network 



The PrePRINT Network ( www.osti.gov/preprint ) was unveiled in January 2000. It is a searchable 
gateway to preprint sites that contain information about scientific and technical disciplines of 
concern to DOE. Such disciplines include physics, materials, chemistry, and portions of biology, 
environmental sciences, and nuclear medicine. Collections and resources included on the 
PrePRINT Network are provided by academic institutions, government research laboratories, 
scientific societies, private research organizations, and individual scientists and researchers. The 
PrePRINT Network facilitates access to these resources, providing a comprehensive set of 
energy research information, but does not change the content or data provided by the originating 
site or author. 

The PrePRINT Network expedites the dissemination of scientists’ research. It is Web-based and 
provides access to energy-related papers, draft journal articles, and other electronic research 
materials at 1,000 preprint sites housing over 330,000 documents. More than twenty preprint 
databases are searchable via a single query. In addition, the PrePRINT Network provides links to 
over 170 related scientific societies and associations. 

Users are offered three search options. A user can browse one specific preprint site or a selected 
set of sites; within this option, the user may also perform an indexed search of the HTML pages 
of the available sites. Alternatively, the user can choose “Search Selected Sites” and search 
multiple preprint sites with a single query. Subject Pathways, the user’s third option, allows the 
user to browse collections by subject area. 

In most cases, access to the full-text information on the target sites is open and free of charge. By 
eliminating the need to locate individual preprint sites through Web searching, researchers can 
find more relevant information while saving time. The PrePRINT Network is a single point of 
entry for preprints in the scientific and technical areas. 

Additional Digital Collections 

In addition to this trilogy of products that addresses the three main ways by which researchers 
disseminate their findings, OSTI has built and developed complementary digital collections, 
including: 

• DOE R&D Project Summaries: brief descriptions of over 17,000 R&D projects 
currently ongoing within the DOE 

• DOE R&D Accomplishments: outcomes of past DOE research and development that 
have had significant economic impact, have improved people’s lives, or have been widely 
recognized as a remarkable advance in science 

• OpenNet: the DOE legacy collection of declassified documents, developed and 
maintained by OSTI for the DOE’s Office of Declassification 

• ECAPs: electronic current awareness publications providing subject-based collections; 
sponsored by DOE Programs 

• Federal R&D Project Summaries: brief descriptions that demonstrate the value of a 
portal to information about federal research projects 
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DOE R&D Project Summaries ( www.osti.gov/rdprojects ), unveiled in June 1997, provides the 
public with access to key corporate information for more than 17,000 research and development 
projects performed since 1995 by the department’s laboratories and other research facilities. It 
includes DOE research activities in a wide variety of energy-related scientific disciplines. It has 
received DOE’s Information Management Quality Award for Management/ Administrative 
Excellence in 1997 and was recognized with a Hammer Award from Vice President Gore’s 
National Partnership for Reinventing Government in 1999. 

DOE R&D Accomplishments ( www.osti.gov/accomplishments ) is a Web site showcasing the 
proud heritage of the department’s research and development and highlights benefits that are 
being realized now. It was unveiled in March 1999 as a central forum for providing the public 
with information about outcomes of past DOE-sponsored or generated research and 
development. The core of the Web site is a searchable database. Complementing the database is 
a page of “Snapshots” that highlight research and development accomplishments. When 
information about a Snapshots topic becomes available from the DOE R&D Accomplishments 
Database, links to full-text reports are identified and provided. 

OpenNet ( www.osti.gov/opennet ) provides easy, timely access to recently declassified DOE 
information, including information declassified in response to Freedom of Information Act 
requests. It includes references to all documents declassified and made publicly available after 
October 1, 1994. 

ECAPs (Electronic Current Awareness Publications) ( www.osti.gov/ecaps ) is a collection of 
bibliographic citations, broken out by subject area, from the Energy Science and Technology 
Database (EDB). For DOE reports, links are provided to full-text documents. These long- 
standing paper publications were recently transitioned to a searchable Web product. OSTI 
publishes several separate ECAPs and maintains a collection of over 30,000 ECAP citations. 

Federal R&D Project Summaries ( www.osti.gov/fedrnd) was released in April 2000 and 
provides a unique window to the federal research community, allowing agencies to better 
understand the research and development efforts of their counterparts in government. It provides 
insight to the public in how its investment in research and development is being used and 
supports full-text single-query searching across databases residing at different governmental 
agencies. 



EnergyFiles 

The umbrella for this suite of resources is EnergyFiles ( www.osti.gov/energyfdes ), which was 
released in May 1997. It is a Web-based virtual library that provides easy access to over 500 
widely diverse collections of both DOE and worldwide energy-related STI. 

The EnergyFiles search mechanism, EnergyPortal Search, provides for increased site efficiency 
and ease of knowledge discovery. EnergyPortal has conquered a major obstacle confronting 
multi-source virtual libraries. Its unique search capability provides distributed searching across 
decentralized, heterogeneous databases and Web sites linked to EnergyFiles. Words or phrases 
are entered in a single query box and the query is distributed in parallel to the user-selected 
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multiple databases and Web sites residing at diverse locations. 

EnergyPortal Search continues to represent a breakthrough in information retrieval. It enables 
users to search across 26 databases and 500 Web sites. The sites, maintained by various agencies, 
are geographically dispersed, and require no standardization in terms of format, software or 
metadata. EnergyPortal will search full text (when available); DOE databases and collections; 
databases of other agencies such as the Defense Technical Information Center (DTIC), the 
National Aeronautics and Space Administration (NASA), the National Library of Medicine 
(NLM) and the Environmental Protection Agency (EPA); and other resources. When the 
individual database supports it, the searched word or phrase is highlighted for easy access. 

Awards and recognitions received by EnergyFiles include: 

• Vice President Gore’s National Performance Review Hammer Award 

• Inclusion in the October 1, 1998 inaugural issue of Access America Online Magazine 

• Favorable review in the University of Wisconsin’s “Scout Report” for science and 
engineering 

• Feature spot in “Federal Computer Week,” with emphasis on EnergyPortal 

Future Information Infrastructure for the Physical Sciences 

This history of success lays the foundation for a future Information Infrastructure for the 
Physical Sciences, which focuses on energy, science, and technology. The goal of this new 
initiative is to provide a comprehensive resource for worldwide scientific information available 
at the desktop, a Web-based network that can be accessed by researchers, engineers, educators, 
students, industry, and the public. 

The objectives of the Information Infrastructure for the Physical Sciences are to deliver a 
permanent, comprehensive resource for accessing and using scientific information; facilitate 
research and discovery to secure a healthy and competitive science and technology future; raise 
scientific and technological literacy of all Americans; produce the finest scientists and engineers 
for the 21 st century; promote scientific research and development results as a foundation for 
future advancements; and establish a digital library that is complementary to existing national 
libraries in providing federally-sponsored information to the public. These existing libraries 
include the National Library of Medicine, the National Agricultural Library, the National Library 
of Education, the National Transportation Library, the EPA National Library Network, and the 
National Science, Mathematics, Engineering, and Technology Education Digital Library 
(NSDL). 

The Information Infrastructure for the Physical Sciences will significantly expand DOE’s local 
presence across the nation. Such an active publicly-oriented presence will bring scientific and 
technical information, energy data and prices, and consumer and educational information to the 
regional level for application and use at the local level by consumers, researchers, educators, 
students, and industry. 

A major challenge for a digital library is to incorporate search capabilities across heterogeneous 
databases and Web sites when there is no standardization of data and information resides in 
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multiple forms on a variety of unrelated systems at widely dispersed facilities. Sophisticated 
distributed searching capabilities will allow the user to access information without having to 
know which database to use, which information collection to pursue, or the organizational 
structure of the agency making the information available. This search capability must be 
augmented by the ability to deliver the information retrieved electronically to the desktop, either 
directly, through licensing agreements, or through other cooperative arrangements. 

The initiative will ensure the delivery of validated research information while strengthening and 
sustaining the nation’s leadership in science and technology. Resource requirements, partnership 
arrangements, and numerous other planning activities are currently being explored as support for 
this initiative continues to grow. 



The Q&A-Web Connection: 

Developing Online Resources for Digital Reference 

Pauline Lynch Shostack 
AskERIC 

Presentation 

Introduction 

This session describes how the AskERIC question-answer service is expanding 
content on its Web site based on questions received. Newly added features, expected 
January 2001, include the AskERIC Response Archive and over 4,000 educational 
resources. 
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The Q&A-Web Connection: 
Developing Online Resources 
for Digital Reference 

Facets of Digital Reference 
October 16, 2000 









AskERIC Background 

■ AskERIC is a federally funded digital 
reference service that specializes in 
responding to requests for educational 
information and providing educational 
resources. 

■ AskERIC receives approximately 40,000 
e-inai I/Web questions a year. Our Web site 1 3 
averages 2.5 million hits per \yee^=|jp^p.-^ 






AskERIC Background 



■ Q&A 

♦ Handle Web specific 
questions 

♦ Entire ERIC system 
participates 

♦ Collect demographics 

♦ Control of Q&A page 
development . 

♦ Development of Q&A 
back end . 



■ Web Site 

♦ Develop content on the 
site 

♦ Monitor lesson plan 
and feedback accounts 

♦ Create policy & 
procedures for the site 

♦ Development of new 

Web site - 
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$ Current Q&A and Web 






Connection 






... 


■ All questions arc routed to one account 




* 


■ Each staff person has Web development responsibilities 

■ Q&A features have expanded due to utilization of 




information specialist expertise 
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♦ Response Archive 

♦ Resource Collection 
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■ 1 ncl us ion of ER 1C System 


W Q&A Service 


• 


staff 
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■ Combined effort to redesign 


a Web Development J»jj 




service and site 
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f Benefits of New Structure 

I ■ Better assessment of site issues due to all 

questions being routed to one account. This has 
led to enhancements of features: 

E ♦ ERIC database 

Z ♦ Lesson plan collection 

■ Q&A staff have a deeper involvement in the 

~ development of the site 

■ Decisions made related to the site are based more 
on questions received 

■ New controlled vocabulary based on questions and 

content .^.7 7 . - 7 ' . : 7/7. 



Conclusion 

Integrating your Web development and 
reference services will provide a truly user 
focused Web site. 



. -.>• . . 



AskERIC on the Web 



www.askeric.org 
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Electronic Data Resources Procurement at CIA 



Blane Ampthor 
Central Intelligence Agency 

Presentation 

Introduction 

For the past several years, the Central Intelligence Agency (CIA) library has made 
an effort to utilize more electronic resources and make more reference resources available 
on the desktops of the agency population. This effort has included obtaining data via CD- 
ROM, subscriptions to electronic resources and, when appropriate, the replacement of 
hardcopy resources with their electronic equivalents. 

In an effort to exercise the library’s role as a leader of open-source information 
procurement and utilization within the CIA, the library has recently created a group called 
the Data Resources Council (DRC). The DRC and its subgroup, the Data Resources 
Council Working Group (DRCWG), seek to be the vehicles within the Directorate of 
Intelligence (DI), the analysis-producing arm of CIA, by which new requests for 
electronic data are evaluated. The purpose of forming the DRC and the DRCWG is to 
evaluate new resources and trends in electronic information, to reduce duplicate 
purchases of data, and to foster cooperation and resource sharing in an effort to save 
money while making as many electronic resources available as possible. 

This presentation explains the business process and workflow of the DRC and 
DRCWG as well as the effectiveness of the procedures and groups in achieving the above 
goals. 
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Background 


Electronic Data Resources 




• Management recognized need to better handle 
electronic material procurement and processing 


Procurement at CIA 




• Data Resources Council (DRC) and Data 
Resources Council Working Group (DRCWG) 
formed in early 1998 

• Efforts primarily aimed at Directorate of 


Blanc Ampthor 




Intelligence (DI), the analysis-producing arm of 


Central Intelligence Agency Library 




CIA 


10/16/00 







DRC Purpose and Mission 

• Identify and remedy gaps in and duplication 
of electronic resources across the 
Directorate of Intelligence (DI) 

• Vet plans for new capabilities involving 
open source exploitation 

- CIA has classified and unclassified resources 
and systems; open source defined as publicly- 
available material 



DRC Purpose and Mission 
(Cont’d.) 

• Establish procurement priorities for open 
source materials 

• Leverage our investments: “Right data to 
the right people'’ 

• Develop corporate plans and strategies for 
enhancing variety of resources available to 
users 

• Oversee integration of open source 
materials in corporate knowledge system 
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DRCWG Mission and Purpose 

• Group charged with implementing polices and 
recommendations of DRC 

• Develop process to identify, procure and deploy 
electronic open source resources 

• Eliminate unnecessary redundancy in D1 open 
source electronic data procurements 

• Identify collection gaps - subject areas for which 
resources are weak - by surveying collections 



Approach to Forming 
DRC and DRCWG 

• Establish process for Dl-wide coordination 
of electronic data resources procurement 

• Maintain an inventory of electronic data 
resources available in the DI 

• Identify offices’ focal point to coordinate 
electronic data resources requirements 

• Establish forum to review procurement 
mechanisms and deployment options early 
in the process 



Business Process and Workflow 




DRC Membership 








• Senior managers from each Dl office, 
computer support staff, data procurement 
office 




(Ssdffis^ GostEastf i ) 




• Membership aims to include representatives 
from stages of entire procurement process 











DRCWG Membership 




DRCWG Membership (Cont’d.) 


• Office representatives - experts in content 
evaluation and office data needs - Electronic 
Resource Officer (ERO) 




• P ub 1 i cati on s p u rc h as i n g offi c e 

• Computer support staff 


• Several librarians deployed to offices as 
reference resource and area experts 






• Library staff - provides support to entire 
agency; experts in information needs of all 
employees 







Role of Library in Workflow 

• Content expertise 

- Library reference personnel located in offices 
for support, resource expertise 

• Broker for interested offices and vendors 

• Provides technical evaluation and support 

• Hosts some procured resources - currently 
has about 50 sources at desktop 

• JDRCWG chair and coordinator 



Database of Requests 

• Numerous issues addressed due to various 
sources of electronic data and user 
requirements 

• Fee-based resources required coordination 
of budget personnel, requester, supervisor 

• Structured review process implemented 

• Notification to requester of each change in 
status during evaluation process 
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Database of Request (Cont’d.) 




Database Issues 


• Categorization of requests by topic and 




• Origin of resource 


geographic region 




• Method of access to resource 


• Some existing records of electronic data 




• Archive needs of users 


previously contracted fed into database 




• Update frequency 

• Language of resource 

• Conditions making resource useful 



Database Issue (Cont’d.) 




Success Stories 


• Method of access 




• DRCWG contacted by groups outside the 


- Hosted on intranet, standalone, Internet 




DI 


subscription 




• Process identified previously existing 


• Conditions making access useful 




resources received as new requests 


detenni nation of needs of user to influence 




• Database has become a repository of current 


pursuit of resource 




electronic resources 


- needs include delivery frequency, full text and 




• Records maintained of requests unable to be 


graphics availability 




filled for later review 




o 

o 
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Success Stories (Cont’d.) 

• Sharing of resources by multiple offices 
with library hosting material 

• DRCWG brokers offices’ share of existing, 
high-cost resources 

• Increasing coordination with other data 
purchasers within CIA 



Contribution to Remote 
Reference 

• Process makes more resources available to 
more users by a common host such as the 
library 

• Offices are less likely to purchase products 
for their exclusive use 

• Process allows existence and applicability 
of products to be advertised in a formal 
fashion 



Contribution to Remote 
Reference (Cont’d.) 

• Library has designated contacts who 
provide input on utility of potential and 
existing products 

• Product evaluation process raises awareness 
of resources 

• Coordination with other groups attempting 
similar process 

• DRC has endorsement of high level 
managers 



Obstacles to Work Flow 

• Offices are not forced to participate - it is in 
the best interest of the office to do so 

• Offices can still purchase products by and 
for themselves 

• DRCWG is not an approval mechanism — 
just an evaluation and recommendation 
body 
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Obstacles to Workflow (Cont’d.) 




Future Roles of DRC and 
DRCWG 


• Offices have varying amounts of money for 




• Expansion outside the Dl 


resources and unique priorities 




* More and continuing marketing as Dl funds 


• Offices have varying levels of need to use 




decrease 


the DRCWG 




♦ Merging of several other existing data 






procurement efforts 

* Forum to evaluate other products associated 
with data resources 



Future Roles of DRC and 
DRCWG (Cont’d.) 

• Act as a broker of funding of data resources 
including possible role of source of some 
funds 
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Evaluating Digital Resources: 

The VIVA Consortium Assesses GaleNet 

Jim Self 

University of Virginia Library 

Presentation 

Introduction 

The Commonwealth of Virginia established a statewide consortium called VIVA 
(The Virtual Library of Virginia) in 1994 to bring electronic resources to the college and 
university libraries of the state. For the first few years of its existence, VIVA was 
primarily concerned with selecting and acquiring new resources, but now the consortium 
faces the need to rigorously evaluate existing resources and to decide which titles should 
be continued. 

This report describes a process used to evaluate one set of digital reference 
resources. VIVA has subscribed to this set since January 1997 and is currently making a 
decision whether to continue the databases for 2001 and future years. The Resources for 
Users Committee is responsible for devising and conducting the evaluation. 

The evaluation instrument includes an examination of the usage statistics for each 
database as compared with its cost. Also, librarians at VIVA institutions have completed 
a Web based survey indicating the importance of each title and noting if their institution 
also has a print subscription to the title. The survey form encouraged librarians to 
comment about each database. 

The qualitative results have been merged into tabular and graphical presentations, 
which display usage and ratings in a single document, allowing easy comparison of the 
databases. The data for each title were examined as a composite, and then analyzed by 
type of institution (doctoral university, four year college, or community college), because 
it is important to know if the results vary significantly among the various types of 
institutions. 

In addition to the qualitative measures, the evaluation group has conducted a more 
traditional qualitative assessment, including a narrative report, which considered the 
coverage, usability and technical performance of each title. The findings of the qualitative 
report and the quantitative data were compared and incorporated into the final report, 
which includes recommendations for the continuation or cancellation of each title. 

This tool was developed to evaluate a single set of related databases, but in the 
future it will serve as a model for evaluating other VIVA databases. Furthermore, we 
expect it may have applications for other consortia and institutions. 
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Evaluating Digital Resources 

The VIVA Consortium Assesses GaleNet 

Unit Costs 
Ratings 

Usage Statistics 
Print Cancellations 
Negotiations 
Recommendations 



Jim Seif 

University of Virginia Library 
VIVA Resources For Users Committee 
Spring, Summer 2000 



Estimated Unit Costs per Search 





Annual 

Cost 


Estimated Cost 
per Search 


Associations 

Unlimited 


$49,500 


$4.41 


Biography and 
Genealogy 
Master Index 
(BGMI) 


$33,000 


$2.47 


Contemporary 

Authors 


$49,500 


$2.47 


Publications & 
Broadcast Media 


$0 


$0 


Research 

Centers 


$17,600 


$10.97 


Composite 


$149,600 


$3.09 



All Public Institutions 
Usage and Ratings 




Public Institutions by Type 
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Private Institutions 
Usage and Ratings 




Print Subscription Status - Public VIVA Institutions 




Contemporary Authors 




Associations Unlimited 


• Highest rating, highest statistics 




♦ Use and rating both reasonably high 


• Low cost per search 




♦ Cost per search is mid-range 


• Crucial database, but online version is flawed 




♦ Used equally by all types of institutions 


• Many libraries have retained the print version 




♦ Over 50% of institutions have cancelled print 


• Recommend renewal, but Gale should address our 
concerns, in writing, before we renew 




♦ Recommend renewal 
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BGMI 

(Biography and Genealogy Master Index) 

• Mid-range usage and rating 

• Important for comprehensives and doctorals 

• Unimportant for community colleges 

• Cost per search is low 

• Most subscribers have cancelled 

• Recommend renewal 



Publications and Broadcast Media 

• Lower ratings and use 

• Most subscribers have cancelled 

• VIVA is receiving title at no charge 

• Recommend renewal as a free or very 
low cost title 



Research Centers Directory 




Negotiations 


• Lower ratings and usage 




• Start the process early 


• High cost per search 




• Identify the issues clearly 


• Questionable as a VIVA product, but 




• State the problems in writing 


Gale is ceasing publication 




• Insist on an authoritative, written response 

• Suspend the process until you get an answer 
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Pricing 




Recommendations for the 
Evaluation Process 


• Recommendations based on assumption 




• Diverse group of evaluators 


that cost per title will not increase 




- By geography and type of institution 


significantly. 




• Varied forms of information 


• Total cost should be no more than the 
current price including Research Centers. 




- Usage statistics 

- Quantitative ratings by librarians 

- Narrative reviews 

- Cancellation data 

- Open ended comments by librarians 



Credits 

Subcommittee members: 

Jim Self, University of Virginia, Chair 
Jacque Dessino, Tidewater CC 
Karen Hartman, Mary Washington College 
Linda Richardson, Virginia Tech 
U.Va. Management Information Services: 
Dave Griles, Programmer 
Joy Borkholder, Intern 
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Facets of 
Mat Reference 

The Virtual Reference Desk 

2nd Annual Digital Reference Conference 

October 16-17, 2000 • Seattle, Washington USA 



Conference Proceedings 

About the Contributors 

A|B|C|D|E|F|G|H|I|J|K|L|M|N|0|P|Q|R|S|T|U|V|W|X|Y|Z 



A 

Jessica Albano received her M.L.I.S. from the University of Washington's School of Library and Information 
Science. She is currently the communications studies librarian at the University of Washington's Suzzallo and 
Allen Libraries. As one of the first librarians to manage LibQuest, Jessica helped to shift the focus to 
incorporating electronic reference as well as electronic information services. 

Blane Ampthor is reference librarian at the Central Intelligence Agency, where he has worked since 1985. 
Ampthor has worked in a variety of library-related jobs at CLA and the library since 1991. Ampthor spent four 
years on loan supporting a geographic office as an on-site librarian. He received his MLS in 1993 from Catholic 
University in Washington, DC and a B.A. in Communications from University of Scranton in 1982. 

Eric Anderson is a library assistant for Electronic Reference Services at North Carolina State University 
Libraries. He worked as a library assistant in the Research and Information Department at NCSU Libraries 
since 1990 and as a student assistant in the Reference Department at NCSU Libraries from 1986-1989. Eric 
received a B.A. in Urban and Regional Planning at Eastern Washington University in 1985. He completed 
graduate coursework in Landscape Architecture at North Carolina State University, and is a graduate student in 
Library and Information Science at North Carolina Central University. 

Meredith Ault is a distance learning librarian at the Florida Distance Learning Reference & Referral Center. 
She graduated with her M.S.L.I.S from the University of North Carolina-Chapel Hill in 1998. She worked as a 
corporate librarian for Unisys Corporation in Philadelphia before joining the Florida Distance Learning 
Reference & Referral Center in March 1999. 
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Silvia Barcellos is a Ph.D. candidate at the School of Information Studies at Syracuse University. Prior to her 
enrollment in the Ph.D. program, she worked for a Brazilian government research institute where she 
participated in the design and implementation of information systems for libraries and information centers. 

Silvia received a B.S. in Electrical Engineering from the University of Brasilia and an M.S. in Computer 
Science from the Instituto Nacional de Pesquisas Espaciais in Sao Paulo. Her dissertation is titled 
Understanding Intermediation in a Digital Environment: An Exploratory Case Study. Silvia plans to defend her 
dissertation by spring 2001. She is a member of the American Society for Information Science (ASIS) and the 
Association for Computing Machinery (ACM). Silvia is the VRD 2000 Conference Student Paper Award 
Winner. 

Blythe Bennett is the Virtual Reference Desk Learning Center Coordinator at the Information Institute of 
Syracuse at Syracuse University. Before joining the VRD project, she coordinated the KidsConnect project, an 
Internet based Q&A project for K-12 students. Blythe was a high school Spanish teacher, sixth grade teacher 
and elementary school librarian. She received her M.L.S. from Syracuse University and her bachelor’s degrees 
from Marietta College. 

Josh Boyer is a reference librarian for distance learning at North Carolina State University Libraries, where he 
began working in August 1999. Prior to library school, he worked as a newspaper reporter in Chapel Hill and 
Tarboro, North Carolina. Boyer received his M.L.S. from the University of North Carolina at Chapel Hill in 
1999 and a B.A. in English from the University of North Carolina at Chapel Hill in 1994. 

Brett Butler is the founder and CEO of AnswerBase Corporation, a new reference information system. 
AnswerBase is creating new standards of reference support, based on an innovative standardized data content 
format. He has been a database publisher, librarian, and bookseller and has spoken and written widely in the 
education community and information industry. 

c 

Karen Ciccone is acting head of research and information services at North Carolina State University Libraries. 
She has also served as reference librarian for Physical and Mathematical Sciences from 1997 to the present. 
Karen received her M.L.S. from University of North Carolina at Chapel Hill in 1997 and her B.S. in Physics 
from Rhodes College in Memphis, TN in 1987. She taught high school physics in Santa Barbara, CA from 
1991-1995. 

Steve Coffman is the new Product Development Manager for LSSI (Library Systems and Services Inc.), an 
organization providing professional library services to a wide variety of institutions including the Department of 
Energy, the Office of Veterans Affairs, Riverside County, Jersey City, Calabasas, Hemet, and Chatham College. 
Steve will focus on developing live, real-time reference services on the Web for the libraries that work with 
LSSI. Prior to coming to LSSI, Steve worked for 15 years at the County of Los Angeles Public Library as the 
Director of FYI, the County’s Business Research Service. Steve has written several articles for library trade 
journals, including “What If You Ran Your Library Like a Bookstore?”, “Building Earth’s Largest Library”, 
“Reference As Others Do It”, “And Now a Word from Our Sponsor”, and, most recently, “The Librarian and 
Mr. Jeeves”, which appeared in the May 2000 issue of American Libraries. Coffman is the recipient of the 2000 
VRD Director’s Award. 

Paul Constantine heads the Reference Services Division of Cornell’s 01in*Kroch*Uris Libraries. He is 
responsible for planning, implementing, managing and evaluating reference and instructional services for the 
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largest library unit on the Cornell campus. One of his chief interests is reference service for remote users; his 
* division has experimented with providing reference service via video-conference software and is currently 
providing live, interactive service using LivePerson software. 

D 

Vera Daugaard is reference librarian at Heming County Library in Denmark, where she has worked since 
receiving her library degree in 1979. Currently, she is the project manager of Net Librarian. 

JoAnn DeVries is reference librarian and bibliographer for the agricultural sciences at the University of 
Minnesota, where she has worked since 1986. She is coordinator of collections for the libraries supporting the 
College of Agriculture, Food and Environmental Sciences, College of Biological Sciences, College of Human 
Ecology, College of Natural Resources, College of Veterinary Medicine, and the School of Social Work. 

E 

Michael B. Eisenberg is director of the Information School at the University of Washington, Seattle. Mike 
conducts research, writes, consults, and lectures frequently on information problem-solving, information 
literacy, information technology, the Internet, and information management in learning and teaching. He is co- 
creator of the Big6 Skills approach to information problem-solving and technology in learning and teaching. 
Mike earned his M.L.S. from the State University of New York at Albany and his Ph.D. in Information Transfer 
from the School of Information Studies at Syracuse University. 

F 



Valerie Footz is an instructional librarian at the Northern Alberta Institute of Technology (NAIT) in Edmonton, 
Alberta. Prior to this position, she was shared between Grant MacEwan College and NAIT to work on the Ask 
A Question project. From 1990 to 1999, she was a law librarian at the Access to Justice Network (Faculty of 
Extension, University of Alberta) which is a Web-based service used to provide legal information to Canadians. 

G 



Melanie A. Gardner is the coordinator of AgNlC at the National Agricultural Library (NAL). She coordinates 
the AgNlC Alliance partnership and maintains communication among partners. Prior to her present position, 
she was the social sciences librarian in NAL's Rural Information Center. Before coming to NAL, she worked 
for the University of Maryland at two different campuses. She holds a B.S. in Education, and a Master of 
Library Science with a concentration in rare books and manuscripts/archives. Ms. Gardner taught school for 7 
years and has been a librarian for over 15 years and has been part of the transition from traditional to electronic 
reference services. 

Suzanne Gray received her master’s degree from the University of Michigan’s School of Information in 1999 
and is currently a member of the University Library’s Residency Program. Her forthcoming publication is an 
analysis of virtual reference services in academic libraries, and she coordinates the University Library’s Ask Us 
virtual reference service. 
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Adam Hall has been the manager of operations for reference and research services in the University of 
- Washington’s Suzzallo and Allen Libraries since 1997, and coordinator of information services since 1992. He 
was assistant head of circulation at the Berkeley Public Libraries for five years prior to moving to the Pacific 
Northwest and joining the UW Libraries staff. Adam was one of the originators of the Libraries’ LibQuest e- 
mail information service and currently continues his involvement. He also has a master’s degree in counseling 
psychology and has been in private counseling practice since 1991. 

Bruce Henson is assistant head of reference at the Georgia Institute of Technology Library and Information 
Center. He received an M.L.S. from the University of North Carolina at Chapel Hill School of Information & 
Library Science. Henson has worked at Georgia Tech since 1998. He coordinates the digital reference service. 

Carol A. Hert is on the faculty of the School of Information Studies at Syracuse University and is a 2000-2001 
American Statistical Association/National Science Foundation/Bureau of Labor Statistics Research Fellow at 
the United States Bureau of Labor Statistics. She has significant research experience in user information seeking 
behavior, system evaluation and design (related to information seeking), and organizational responses to 
technology. Her current funded research concerns statistical agency efforts to address the needs of citizens. 

Chrystie Hill is a second year graduate student with the University of Washington Information School and a 
student reference librarian for the Seattle Public Library. Her primary research interests are in reference, 
particularly the reference exchange in digital environments. As a research assistant for the Information School, 
she worked in 1999-2000 with Joe Janes and Alex Rolfe on a grant from the Library of Congress characterizing 
attitudes about digital reference and digital reference services in public and academic libraries, and analyzing 
information exchange services on the World Wide Web. Their current research characterizes commercial and 
non-commercial Ask An Expert services and will be published after further development in the following 
academic year. 

J 

Joseph Janes is assistant professor at the Information School of the University of Washington. He is interested 
in reference, particularly in the use of technologies to mediate and assist, and the use of networked resources in 
reference. His research is on models of practice in digital reference. He teaches courses in reference, online 
searching, research methods and statistics, and on the use of Internet technologies in librarianship. Janes holds 
an M.L.S. and Ph.D. from Syracuse University. 

Margo Jeske is a reference librarian with the Library of Parliament in Ottawa, Canada. Margo graduated from 
the University of Western Ontario’s Master of Library and Information Science program in 1983. Since joining 
the Library of Parliament in 1989, Margo has worked as both a reference librarian and database coordinator, 
and has been involved with the PARLREF project since 1997. 

K 

Cindy Kaag is head of the Sciences Libraries at Washington State University. She has been at Washington 
State University since 1986. She has served previously as head of collection development for the Science and 
Engineering Library and head of the Education and Agricultural Sciences Libraries. She is involved with 
collection evaluation and participated in a grant to encourage sharing of resources in the Pacific Northwest; she 
also published an annotated bibliography of collection evaluation techniques through ALA. She teaches an 
information literacy class to distance education students in agriculture. 

Diane Kresh is director for public service collections and director for preservation at the Library of Congress 
(LC). Her experience at LC has covered a wide spectrum of responsibilities, including copyright, collections 
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maintenance, document delivery, reference, user training, photo duplication services, and conservation. Diane . 
v serves on the LC Internet Policy Committee, directs nearly all the library’s general and special collections, and 
is leading the effort to provide enhanced Internet services for public researchers. 

L 

R. David Lankes is director of the Information Institute of Syracuse (IIS) and an assistant professor at Syracuse 
University’s School of Information Studies. Lankes is co-founder of AskERIC, and founder of the Virtual 
Reference Desk Project. Dave speaks and consults nationally on Internet issues in education and business. 
Lankes’ research is in education information and digital reference services. Lankes received his B.F.A. in 
Multimedia Design, M.S. in Telecommunications and Ph.D. in Information Transfer from Syracuse University. 

M 

Barbara MacAdam is currently the head of reference for the University of Michigan Library, and former head 
of the Undergraduate Library at UM. As an adjunct faculty member in the School of Information and the 
College of Literature, Science and Arts, her teaching credits include courses in information resources and 
services, user instruction, and an undergraduate seminar for honors students on knowledge and society in the 
information age. She has numerous publications in the areas of management, user instruction, undergraduate 
education, and critical thinking, including a co-authored book, Reaching a Multicultural Student Community, 
published by Greenwood Press. 

Michael McClennan is the head of systems for the Internet Public Library and is the developer of QRC, a 
software package for coordinating the operation of digital reference services. He has a Ph.D. in Computer 
Science from the University of Michigan. 

Charles R. McClure is Francis Eppes Professor at Florida State University’s School of Information Studies 
and director of the Information Use Management and Policy Institute. Chuck received the American Library 
Association’s Jesse H. Shera award for the best research study in library/information science for 1990 for his 
study of Electronic Networks, the Research Process, and Scholarly Communication. He has authored and edited 
more than 35 monographs and 250 reports, chapters, and papers. Chuck received his Ph.D. in Library and 
Information Services from Rutgers University. 

Susan McGlamery is the reference coordinator for the Metropolitan Cooperative Library System, and has an 
extensive history managing reference services in legal, corporate and public library environments. 

Joseph Meloche has taught and researched in the area of information seeking for the past several years. He is 
currently working as a lecturer at the School of Information at Charles Sturt University (Australia) in a library 
studies program taught in the distance mode. He also has experience working as a senior subject librarian and 
CD-ROM Coordinator at the University of Sydney’s Fisher Library. He is the author of several papers on 
information seeking and information literacy, as well as the monograph, Introductory CD ROM Searching: The 
Key to Effective Ondisc Searching, Haworth Press, 1994. 

Makiko Miwa is an information consultant for Epoch Research Corporation, a Tokyo-based consulting firm in 
the field of information services. She began working there as an information broker in 1983. Her work focuses 
on user-based research and development of information retrieval systems and services, international information 
transfer, and human information behavior. She also serves on national and industrial IT-related committees in 
Japan. Makiko conducts research, writes, consults, and lectures frequently on information technology, 
information services, and system development. Makiko received her B.A. from Japan Women’s University at 



Tokyo and an M.L.S. from University of Pittsburgh. She received a Ph.D. in information transfer from Syracuse 
* University in 2000. 

o 

Lorena O’English received her M.L.I.S. from the University of Washington’s School of Library and 
Information Science. She has been working as a reference librarian at the University of Washington’s Suzzallo 
and Allen Libraries but will shortly be moving to the Washington State University library system. Her interest 
in using technology to solve information problems began while she was employed at Project Vote Smart, a 
nonprofit political information organization and has grown as she moved into the library field. 

P 

Kit Pitkin - no biography available 

R 

Donna Reed coordinates the Community Information Program at Multnomah County Library in Portland, OR. 
In this capacity, she oversees the development of the library’s Web site, coordinates CascadeLink, a regional 
community information network, and works with governmental Web builders countywide. She received her 
M.L.I.S. from Emporia State University. 

Alex Rolfe is a second year student at the University of Washington Information School. Prior to entering 
library school, he received a masters’ degree in medieval history. Last year, he worked as a research assistant 
for Joe Janes, while learning the ropes as a reference assistant in both the Odegaard Undergraduate Library and 
Government Publications. His current interests are in library history and reference. 

S 

Mary V. Schorn is a technical information specialist for the U.S. Department of Energy (DOE) Office of 
Scientific and Technical Information (OSTI). Among her responsibilities are the management and development 
of two Web-based databases, liaison for distribution of DOE and worldwide energy-related information, and 
special projects. During her employment with OSTI, she has directed and provided leadership for a design 
requirements team for the Energy Files prototype and an infrastructure redesign procedures and training team. 
She has independently developed and maintained the OSTI Government Information Locator Service (GELS) 
records. Schom holds a B.S. from the University of Alabama and a Masters of Science in Library Science from 
the University of Tennessee. 

Joe Schumacher - no biography available 

Roseanne Schwartz - no biography available 

Jim Self is director of management information services for the University of Virginia Library. Since 1996, he 
has served as Chair or Co-Chair of the Collections Group and as Virginia’s collection development 
representative at ALA. Since 1997, Self has served on the The Virtual Library of Virginia (VIVA) Collections 
Committee and its successor, the Resources for Users Committee. In 1998 he co-chaired the evaluation and RFP 
subcommittee for full-text databases and is currently the chair of an evaluation group. Self previously served as 
director of the Clemons Library at the University of Virginia and a librarian at Indiana University in 
^"omington. Self has presented at meetings of the American Library Association, the Virginia Library 



Association, and the Indiana Library Association and has been published in Library and Information Science 
*“ Research, Collection Management, Information Technology and Libraries, and Wilson Library Bulletin. 

Doris Settles is the clearinghouse coordinator for the Center for School Safety. She produces and administers 
print and Web-based publications, an AskA service, a resource library and directory, and a Web site. She is a 
former journalist and teacher of English composition. She holds degrees in English, Journalism and 
Instructional Design from the University of Kentucky. 

Pauline Lynch Shostack coordinates the AskERIC service and its team of information specialists. AskERIC 
Q&A is an electronic service providing research assistance to anyone interested in the process or practice of 
education. The AskERIC Web site provides a wealth of education-related information including a lesson plan 
collection and the ERIC Database. Pauline earned her B.S. in psychology and elementary teacher certification 
from LeMoyne College and received her M.L.S. at Syracuse University. Pauline also works part-time as a 
reference librarian at the Liverpool Public Library and is an adjunct faculty member for the School of 
Information Studies. 



Dr. Joanne Silverstein is head of research and development at the Information Institute of Syracuse and 
assistant research professor at Syracuse University’s School of Information Studies. Her research includes the 
management of e-mail centers and customer communications in federal agencies. She recently completed an 
analysis of e-mail help centers for the Department of Education’s National Library of Education. Silverstein has 
worked in software design, testing and management at General Electric Corporation and Genigraphics 
Corporation, and has consulted with several corporations to harness the Internet and the Web for competitive 
advantage. Silverstein received her Ph.D. in Electronic Commerce and her M.S. in Information Resource 
Management from Syracuse University. 

Karen J. Spence is assistant director for the Office of Program Integration for the U.S. Department of Energy 
Office of Scientific and Technical Information (OSTI). She provides leadership and coordination for the 
department-wide Scientific and Technical Information Program and assures access to the energy-related 
information, which supports the Department of Energy mission. Her duties include coordination with 
departmental organizations, other government agencies, and private and domestic organizations, as well as 
coordination of international partnerships. Ms. Spence is also responsible for OSTI strategic and operational 
planning, policy development, product management, promotions/marketing, and customer advocacy. She has 
more than fourteen years of government and private industry experience. She holds a master’s degree in library 
science from the University of Alabama. 

Tim Steury is director of Ask Dr. Universe, a newspaper and Web service that draws on the resources of the 
research university to answer questions for children of all ages. He is currently the editor of Universe magazine, 
Washington State University’s magazine of research, scholarship, and the arts. He taught literature and writing 
at a number of institutions including the University of Idaho for 1 1 years. He was editor for several years of the 
former Palouse Journal, a regional magazine, and has written on a wide range of subjects for many publications. 



Sam Stormont is electronic reference services librarian and communications subject specialist at Temple 
University Libraries. He was a user services supervisor at Telebase Systems from 1988-1991. He served as 
president of the Academic Assembly of Librarians at Temple University and president of the Drexel University 
College of Information Science & Technology Alumni Association from 1999-2000. He received his B.A. from 
DePauw University, his M.A. from Temple University, and his M.S. from Drexel University. 



Stuart Sutton is associate professor of The Information School of the University of Washington. He teaches in 
the areas of information law and policy, legal informatics, and the organization of information. Sutton received 
his M.L.I.S. in 1986 and Ph.D. in 1991 from the University of California at Berkeley School of Library and 
Information Studies. He is a member of the California Bar and received his J.D. in 1981 from Golden Gate 
O versity School of Law and his LL.M. in copyright from the University of California at Berkeley, Boalt Hall 



School of Law. He holds B.A. and M.A. degrees in theatre arts from San Francisco State University. He has 
* served on the faculties of Syracuse University and San Jose State University and in a visiting position at the 
University of California at Berkeley. 

V 

Rachel Viggiano is a distance learning librarian at the Florida Distance Learning Reference & Referral Center. 
She received her Master of Arts in Library Science from the University of South Florida in 1998. 



w 



David Ward is reference services coordinator at the University of Illinois Undergraduate Library. He worked 
as a graduate assistant at the Undergraduate Library from 1996-1998. Ward has also worked as a 
reference/instruction librarian at the University of Nevada at Las Vegas. His primary responsibilities include 
training for and supervising of the library’s reference desk. His research focuses on using networked technology 
to improve both user services and faculty/librarian productivity. 

Susan Ware is a reference and instruction librarian at Penn State - Delaware County Campus. She has 
experience in general reference, online and chat reference, and Web-based tutorials. She recently co-authored 
the article “Interactive Reference at a Distance: A Corporate Model for Academic Libraries” in The Reference 
Librarian. 

Kate Whitridge is a reference librarian with the Library of Parliament in Ottawa, Canada. Kate graduated from 
the University of Western Ontario’s Master of Library and Information Science program in 1991, and has 
worked for the Library of Parliament for nine years. Assigned to the team charged with identifying a reference 
request management system in 1995, Kate’s time has been dedicated to the in-house development and 
implementation of the PARLREF system for the past year. 

Y 

Joseph Yue is a reference and instruction librarian at the Central Reference Department of the University 
Libraries at the University of Colorado at Boulder. He is interested in the design and implementation of user- 
centered reference services and instruction strategies. 
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Abby S. Kasowitz 

Abby Kasowitz is Coordinator of the Virtual Reference Desk (VRD) Project, a project of 
the United States Department of Education, operated at the Information Institute of 
Syracuse (IIS). Kasowitz is co-author of the AskA Starter Kit: How to Build and Maintain 
Digital Reference Services and co-editor of Digital Reference Service in the New 
Millennium: Planning, Management, and Evaluation. Kasowitz organizes the annual 
VRD conference, an international conference on Internet-based reference service for 
library and information professionals; researches the digital reference field; and assists 
organizations in building digital reference services. 

Kasowitz received an M.L.S. and an M.S. in Instructional Design, Development and 
Evaluation from Syracuse University and a B.A. in English and American Literature from 
Brandeis University. She joined IIS in 1997 as an assistant with KidsConnect, an 
Internet-based question/answer and referral service for K-12 students. 

Joan Stahl 

Joan Stahl is Administrator of Electronic Resources and Image Collections at the 
Smithsonian American Art Museum. She began “Joan of Art” 

(http://AmericanArt.si.edu) . the museum’s digital reference service, in 1993 on America 
Online. The service answers questions received from around the world on the subject of 
American art and artists. 

Stahl received an M.L S. from Rutgers University and an M.A. in Art History from the 
University of Maryland. She has worked in arts librarianship in both public and special 
libraries. She reviews arts publications for a variety of library and educational journals 
and is the Review Editor for Art Documentation, the journal of the Art Libraries Society 
of North America (ARLIS). 
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